Logical and physical ordering

Messages on queues can occur (within each priority level) in physical or logical order.

Physical order is the order in which messages arrive on a queue. Logical order is when all of the messages and segments within a group are in their logical sequence, next to each other, in the position determined by the physical position of the first item belonging to the group.

For a description of groups, messages, and segments, see Message groups. These physical and logical orders can differ because:

  • Groups can arrive at a destination at similar times from different applications, therefore losing any distinct physical order.
  • Even within a single group, messages can get out of order because of rerouting or delay of some of the messages in the group.

For example, the logical order might look like Figure Figure 1:

Figure 1. Logical order on a queue

These messages would occur in the following logical order on a queue:
  1. Message A (not in a group)
  2. Logical message 1 of group Y
  3. Logical message 2 of group Y
  4. Segment 1 of (last) logical message 3 of group Y
  5. (Last) segment 2 of (last) logical message 3 of group Y
  6. Logical message 1 of group Z
  7. (Last) logical message 2 of group Z
  8. Message B (not in a group)
The physical order, however, might be entirely different. The physical position of the first item within each group determines the logical position of the whole group. For example, if groups Y and Z arrived at similar times, and message 2 of group Z overtook message 1 of the same group, the physical order would look like Figure Figure 2:
Figure 2. Physical order on a queue

These messages occur in the following physical order on the queue:
  1. Message A (not in a group)
  2. Logical message 1 of group Y
  3. Logical message 2 of group Z
  4. Logical message 2 of group Y
  5. Segment 1 of (last) logical message 3 of group Y
  6. (Last) segment 2 of (last) logical message 3 of group Y
  7. Logical message 1 of group Z
  8. Message B (not in a group)
Note: On IBM MQ for z/OS®, the physical order of messages on the queue is not guaranteed if the queue is indexed by GROUPID.

When getting messages, we can specify MQGMO_LOGICAL_ORDER to retrieve messages in logical order rather than physical order.

If you issue an MQGET call with MQGMO_BROWSE_FIRST and MQGMO_LOGICAL_ORDER, subsequent MQGET calls with MQGMO_BROWSE_NEXT must also specify MQGMO_LOGICAL_ORDER. Conversely, if the MQGET with MQGMO_BROWSE_FIRST does not specify MQGMO_LOGICAL_ORDER, neither must the following MQGETs with MQGMO_BROWSE_NEXT.

The group and segment information that the queue manager retains for MQGET calls that browse messages on the queue is separate from the group and segment information that the queue manager retains for MQGET calls that remove messages from the queue. When you specify MQGMO_BROWSE_FIRST, the queue manager ignores the group and segment information for browsing, and scans the queue as though there were no current group and no current logical message.

Note: Do not use an MQGET call to browse beyond the end of a message group (or logical message not in a group) without specifying MQGMO_LOGICAL_ORDER. For example, if the last message in the group precedes the first message in the group on the queue, using MQGMO_BROWSE_NEXT to browse beyond the end of the group, specifying MQMO_MATCH_MSG_SEQ_NUMBER with MsgSeqNumber set to 1 (to find the first message of the next group) returns again the first message in the group already browsed. This could happen immediately, or a number of MQGET calls later (if there are intervening groups). Avoid the possibility of an infinite loop by opening the queue twice for browse:

  • Use the first handle to browse only the first message in each group.
  • Use the second handle to browse only the messages within a specific group.
  • Use the MQMO_* options to move the second browse cursor to the position of the first browse cursor, before browsing the messages in the group.
  • Do not use the MQGMO_BROWSE_NEXT browse beyond the end of a group.
For further information about this, see MQGET, MQMD, and Rules for validating MQI options.

For most applications you will probably choose either logical or physical ordering when browsing. However, if you want to switch between these modes, remember that when you first issue a browse with MQGMO_LOGICAL_ORDER, your position within the logical sequence is established.

If the first item within the group is not present at this time, the group that you are in is not considered to be part of the logical sequence.

Once the browse cursor is within a group, it can continue within the same group, even if the first message is removed. Initially though, we can never move into a group using MQGMO_LOGICAL_ORDER where the first item is not present.

    MQPMO_LOGICAL_ORDER
    The MQPMO option tells the queue manager how the application puts messages in groups and segments of logical messages. It can be specified only on the MQPUT call; it is not valid on the MQPUT1 call. If MQPMO_LOGICAL_ORDER is specified, it indicates that the application uses successive MQPUT calls to:
    1. Put the segments in each logical message in the order of increasing segment offset, starting from 0, with no gaps.
    2. Put all the segments in one logical message before putting the segments in the next logical message.
    3. Put the logical messages in each message group in the order of increasing message sequence number, starting from 1, with no gaps. IBM MQ increments the message sequence number automatically.
    4. Put all the logical messages in one message group before putting logical messages in the next message group.

    Because the application has told the queue manager how it puts messages in groups and segments of logical messages, the application does not have to maintain and update the group and segment information about each MQPUT call, because the queue manager maintains and updates this information. Specifically, it means that the application does not need to set the GroupId, MsgSeqNumber, and Offset fields in MQMD, because the queue manager sets these fields to the appropriate values. The application must only set the MsgFlags field in MQMD, to indicate when messages belong to groups or are segments of logical messages, and to indicate the last message in a group or last segment of a logical message.

    After a message group or logical message has been started, subsequent MQPUT calls must specify the appropriate MQMF_* flags in MsgFlags in MQMD. If the application tries to put a message that is not in a group when there is an unterminated message group, or put a message that is not a segment when there is an unterminated logical message, the call fails with reason code MQRC_INCOMPLETE_GROUP or MQRC_INCOMPLETE_MSG, as appropriate. However, the queue manager retains the information about the current message group or current logical message, and the application can terminate them by sending a message (possibly with no application message data) specifying MQMF_LAST_MSG_IN_GROUP or MQMF_LAST_SEGMENT as appropriate, before reissuing the MQPUT call to put the message that is not in the group or not a segment.

    Figure 2 shows the combinations of options and flags that are valid, and the values of the GroupId, MsgSeqNumber, and Offset fields that the queue manager uses in each case. Combinations of options and flags that are not shown in the table are not valid. The columns in the table have the following meanings; Either means Yes or No:

      LOG ORD
      Whether the MQPMO_LOGICAL_ORDER option is specified on the call.

      MIG
      Whether the MQMF_MSG_IN_GROUP or MQMF_LAST_MSG_IN_GROUP option is specified on the call.

      SEG
      Whether the MQMF_SEGMENT or MQMF_LAST_SEGMENT option is specified on the call.

      SEG OK
      Whether the MQMF_SEGMENTATION_ALLOWED option is specified on the call.

      Cur grp
      Whether a current message group exists before the call.

      Cur log msg
      Whether a current logical message exists before the call.

      Other columns
      Show the values that the queue manager uses. Previous denotes the value used for the field in the previous message for the queue handle.

    Table 1. MQPUT options relating to messages in groups and segments of logical messages
    Options you specify Options you specify Options you specify Options you specify Group and log-msg status before call Group and log-msg status before call Values the queue manager uses Values the queue manager uses Values the queue manager uses
    LOG ORD MIG SEG SEG OK Cur grp Cur log msg GroupId MsgSeqNumber Offset
    Yes No No No No No MQGI_NONE 1 0
    Yes No No Yes No No New group id 1 0
    Yes No Yes Either No No New group id 1 0
    Yes No Yes Either No Yes Previous group id 1 Previous offset + previous segment length
    Yes Yes Either Either No No New group id 1 0
    Yes Yes Either Either Yes No Previous group id Previous sequence number + 1 0
    Yes Yes Yes Either Yes Yes Previous group id Previous sequence number Previous offset + previous segment length
    No No No No Either Either MQGI_NONE 1 0
    No No No Yes Either Either New group ID if MQGI_NONE, else value in field 1 0
    No No Yes Either Either Either New group ID if MQGI_NONE, else value in field 1 Value in field
    No Yes No Either Either Either New group ID if MQGI_NONE, else value in field Value in field 0
    No Yes Yes Either Either Either New group ID if MQGI_NONE, else value in field Value in field Value in field
    Note:

    • MQPMO_LOGICAL_ORDER is not valid on the MQPUT1 call.
    • For the MsgId field, the queue manager generates a new message identifier if MQPMO_NEW_MSG_ID or MQMI_NONE is specified, and uses the value in the field otherwise.
    • For the CorrelId field, the queue manager generates a new correlation identifier if MQPMO_NEW_CORREL_ID is specified, and uses the value in the field otherwise.

    When you specify MQPMO_LOGICAL_ORDER, the queue manager requires that all messages in a group and segments in a logical message are put with the same value in the Persistence field in MQMD, that is, all must be persistent, or all must be nonpersistent. If this condition is not satisfied, the MQPUT call fails with reason code MQRC_INCONSISTENT_PERSISTENCE.

    The MQPMO_LOGICAL_ORDER option affects units of work as follows:

    • If the first physical message in a group or logical message is put within a unit of work, all the other physical messages in the group or logical message must be put within a unit of work, if the same queue handle is used. However, they do not need to be put within the same unit of work, allowing a message group or logical message that consists of many physical messages to be split across two or more consecutive units of work for the queue handle.
    • If the first physical message in a group or logical message is not put within a unit of work, none of the other physical messages in the group or logical message can be put within a unit of work, if the same queue handle is used.
    If these conditions are not satisfied, the MQPUT call fails with reason code MQRC_INCONSISTENT_UOW.

    When MQPMO_LOGICAL_ORDER is specified, the MQMD supplied on the MQPUT call must not be less than MQMD_VERSION_2. If this condition is not satisfied, the call fails with reason code MQRC_WRONG_MD_VERSION.

    If MQPMO_LOGICAL_ORDER is not specified, messages in groups and segments of logical messages can be put in any order, and it is not necessary to put complete message groups or complete logical messages. It is the responsibility of the application to ensure that the GroupId, MsgSeqNumber, Offset, and MsgFlags fields have appropriate values.

    Use this technique to restart a message group or logical message in the middle, after a system failure has occurred. When the system restarts, the application can set the GroupId, MsgSeqNumber, Offset, MsgFlags, and Persistence fields to the appropriate values, and then issue the MQPUT call with MQPMO_SYNCPOINT or MQPMO_NO_SYNCPOINT set as required, but without specifying MQPMO_LOGICAL_ORDER. If this call is successful, the queue manager retains the group and segment information, and subsequent MQPUT calls using that queue handle can specify MQPMO_LOGICAL_ORDER as normal.

    The group and segment information that the queue manager retains for the MQPUT call is separate from the group and segment information that it retains for the MQGET call.

    For any given queue handle, the application can mix MQPUT calls which specify MQPMO_LOGICAL_ORDER with MQPUT calls that do not, but note the following points:

    • If MQPMO_LOGICAL_ORDER is not specified, each successful MQPUT call causes the queue manager to set the group and segment information for the queue handle to the values specified by the application, replacing the existing group and segment information retained by the queue manager for the queue handle.
    • If MQPMO_LOGICAL_ORDER is not specified, the call does not fail if there is a current message group or logical message; the call might succeed with an MQCC_WARNING completion code. Table 2 shows the various cases that can arise. In these cases, if the completion code is not MQCC_OK, the reason code is one of the following (as appropriate):

      • MQRC_INCOMPLETE_GROUP
      • MQRC_INCOMPLETE_MSG
      • MQRC_INCONSISTENT_PERSISTENCE
      • MQRC_INCONSISTENT_UOW
      Note: The queue manager does not check the group and segment information for the MQPUT1 call.
    Table 2. Outcome when MQPUT or MQCLOSE call is not consistent with group and segment information
    Current call is Previous call was MQPUT with MQPMO_LOGICAL_ORDER Previous call was MQPUT without MQPMO_LOGICAL_ORDER
    MQPUT with MQPMO_LOGICAL_ORDER MQCC_FAILED MQCC_FAILED
    MQPUT without MQPMO_LOGICAL_ORDER MQCC_WARNING MQCC_OK
    MQCLOSE with an unterminated group or logical message MQCC_WARNING MQCC_OK

    For applications that put messages and segments in logical order, specify MQPMO_LOGICAL_ORDER, as it is the simplest option to use. This option relieves the application of the need to manage the group and segment information, because the queue manager manages that information. However, specialized applications might need more control than that provided by the MQPMO_LOGICAL_ORDER option, which can be achieved by not specifying that option; if we do so, you must ensure that the GroupId, MsgSeqNumber, Offset, and MsgFlags fields in MQMD are set correctly, before each MQPUT or MQPUT1 call.

    For example, an application that wants to forward physical messages that it receives, without regard for whether those messages are in groups or segments of logical messages, must not specify MQPMO_LOGICAL_ORDER, for two reasons:

    • If the messages are retrieved and put in order, specifying MQPMO_LOGICAL_ORDER assigns a new group identifier to the messages, which might make it difficult or impossible for the originator of the messages to correlate any reply or report messages that result from the message group.
    • In a complex network with multiple paths between sending and receiving queue managers, the physical messages might arrive out of order. By not specifying MQPMO_LOGICAL_ORDER and MQGMO_LOGICAL_ORDER on the MQGET call, the forwarding application can retrieve and forward each physical message as soon as it arrives, without waiting for the next one in logical order to arrive.

    Applications that generate report messages for messages in groups or segments of logical messages must also not specify MQPMO_LOGICAL_ORDER when putting the report message.

    MQPMO_LOGICAL_ORDER can be specified with any of the other MQPMO_* options.


Putting Logically Ordered Groups to a Clustered Queue (MQOO_BIND_ON_GROUP)

The MQOO_BIND_ON_OPEN option ensures that all messages from this application, and therefore all groups, are routed to a single instance. This has the drawback that the application traffic is not load balanced across multiple instances of a cluster queue. In order to enable workload balancing while keeping groups of messages intact, you must set the following options:

  • The MQPUT call must specify MQPMO_LOGICAL_ORDER
  • The MQOPEN call must specify one of the following two options:

    • MQOO_BIND_ON_GROUP
    • MQOO_BIND_AS_Q_DEF, and the queue definition must specify DEFBIND(GROUP)

Workload balancing is then driven between groups of messages without requiring an MQCLOSE and MQOPEN of the queue. Between groups means that MQMF_MSG_IN_GROUP is set in the MQMD(v2) or MQMDE, and there is no partially complete group in progress. When a group is in progress, the resolved queue manager and queue name in the object handle are reused.

If the previous message was MQPMO_LOGICAL_ORDER and/or MQMF_MSG_IN_GROUP was set but the current message is not part of the group, then the PUT call fails with MQRC_INCOMPLETE_GROUP.

If an individual MQPUT does not specify MQPMO_LOGICAL_ORDER, and no current group is active, then workload balancing is driven for that message (as if the MQOPEN call has specified MQOO_BIND_NOT_FIXED).

No reallocation takes place for messages bound to a destination using MQOO_BIND_ON_GROUP. For more information on reallocation, see Message groups.