The order in which messages are retrieved from a queue

 

You have control over the order in which you retrieve messages from a queue. This section looks at the options.

 

Priority

A program can assign a priority to a message when it puts the message on a queue (see Message priorities). Messages of equal priority are stored in a queue in order of arrival, not the order in which they are committed.

The queue manager maintains queues either in strict FIFO (first in, first out) sequence, or in FIFO within priority sequence. This depends on the setting of the MsgDeliverySequence attribute of the queue. When a message arrives on a queue, it is inserted immediately following the last message that has the same priority.

Programs can either get the first message from a queue, or they can get a particular message from a queue, ignoring the priority of those messages. For example, a program may want to process the reply to a particular message that it sent earlier. For more information, see Getting a particular message.

If an application puts a sequence of messages on a queue, another application can retrieve those messages in the same order that they were put, provided:

  • The messages all have the same priority

  • The messages were all put within the same unit of work, or all put outside a unit of work

  • The queue is local to the putting application

If these conditions are not met, and the applications depend on the messages being retrieved in a certain order, the applications must either include sequencing information in the message data, or establish a means of acknowledging receipt of a message before the next one is sent.

On WebSphere MQ for z/OS, the queue attribute, IndexType, can be used to increase the speed of MQGET operations on the queue. For more information, see Type of index.

 

Logical and physical ordering

Messages on queues can occur (within each priority level) in physical or logical order. Physical order is the order in which messages arrive on a queue. Logical order is when all of the messages and segments within a group are in their logical sequence, adjacent to each other, in the position determined by the physical position of the first item belonging to the group.

For a description of groups, messages, and segments, see Message groups. These physical and logical orders may differ because:

  • Groups can arrive at a destination at similar times from different applications, therefore losing any distinct physical order.

  • Even within a single group, messages may get out of order due to rerouting or delay of some of the messages in the group.

For example, the logical order might look like Figure Figure 10:

Figure 10. Logical order on a queue

These messages would appear in the following logical order on a queue:

  1. Message A (not in a group)

  2. Logical message 1 of group Y

  3. Logical message 2 of group Y

  4. Segment 1 of (last) logical message 3 of group Y

  5. (Last) segment 2 of (last) logical message 3 of group Y

  6. Logical message 1 of group Z

  7. (Last) logical message 2 of group Z

  8. Message B (not in a group)

The physical order, however, might be entirely different. As stated in topic "Logical and physical ordering", the physical position of the first item within each group determines the logical position of the whole group.

These messages appear in the following physical order on the queue:

  1. Message A (not in a group)

  2. Logical message 1 of group Y

  3. Logical message 2 of group Z

  4. Logical message 2 of group Y

  5. Segment 1 of (last) logical message 3 of group Y

  6. (Last) segment 2 of (last) logical message 3 of group Y

  7. Logical message 1 of group Z

  8. Message B (not in a group)

Note:
On WebSphere MQ for z/OS, the physical order of messages on the queue is not guaranteed if the queue is indexed by GROUPID.

When getting messages, you can specify MQGMO_LOGICAL_ORDER to retrieve messages in logical rather than physical order.

If you issue an MQGET call with MQGMO_BROWSE_FIRST and MQGMO_LOGICAL_ORDER, subsequent MQGET calls with MQGMO_BROWSE_NEXT must also specify this option. Conversely, if the MQGET with MQGMO_BROWSE_FIRST does not specify MQGMO_LOGICAL_ORDER, neither must the following MQGETs with MQGMO_BROWSE_NEXT.

The group and segment information that the queue manager retains for MQGET calls that browse messages on the queue is separate from the group and segment information that the queue manager retains for MQGET calls that remove messages from the queue. When MQGMO_BROWSE_FIRST is specified, the queue manager ignores the group and segment information for browsing, and scans the queue as though there were no current group and no current logical message.

Note:
Special care is needed if an MQGET call is used to browse beyond the end of a message group (or logical message not in a group) when MQGMO_LOGICAL_ORDER is not specified. For example, if the last message in the group happens to precede the first message in the group on the queue, using MQGMO_BROWSE_NEXT to browse beyond the end of the group, specifying MQMO_MATCH_MSG_SEQ_NUMBER with MsgSeqNumber set to 1 (to find the first message of the next group) would return again the first message in the group already browsed. This could happen immediately, or a number of MQGET calls later (if there are intervening groups).

The possibility of an infinite loop can be avoided by opening the queue twice for browse:

  • Use the first handle to browse only the first message in each group.

  • Use the second handle to browse only the messages within a specific group.

  • Use the MQMO_* options to move the second browse cursor to the position of the first browse cursor, before browsing the messages in the group.

  • Do not use the MQGMO_BROWSE_NEXT browse beyond the end of a group.

For further information about this, see WebSphere MQ Application Programming Reference.

For most applications you will probably choose either logical or physical ordering when browsing. However, if you want to switch between these modes, remember that when you first issue a browse with MQGMO_LOGICAL_ORDER, your position within the logical sequence is established.

If the first item within the group is not present at this time, the group you are in is not considered to be part of the logical sequence.

Once the browse cursor is within a group, it can continue within the same group, even if the first message is removed. Initially though, you can never move into a group using MQGMO_LOGICAL_ORDER where the first item is not present.

 

Grouping logical messages

There are two main reasons for using logical messages in a group:

  • The messages may need to be processed in the correct order

  • Each of the messages in a group may need to be processed in a related way.

In either case, retrieval of the entire group must be carried out by the same getting application instance.

For example, assume that the group consists of four logical messages. The putting application looks like this: 

   PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT
 
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP
 
   MQCMIT

The getting application chooses not to start processing any group until all of the messages within it have arrived. MQGMO_ALL_MSGS_AVAILABLE is therefore specified for the first message in the group; the option is ignored for subsequent messages within the group.

Once the first logical message of the group is retrieved, MQGMO_LOGICAL_ORDER is used to ensure that the remaining logical messages of the group are retrieved in order.

So, the getting application looks like this: 

   /* Wait for the first message in a group, or a message not in a group */
   GMO.Options = MQGMO_SYNCPOINT | MQGMO_WAIT
               | MQGMO_ALL_MSGS_AVAILABLE | MQGMO_LOGICAL_ORDER
   do while ( GroupStatus == MQGS_MSG_IN_GROUP )
      MQGET
      /* Process each remaining message in the group */
      ...
 
   MQCMIT

For further examples of grouping messages, see Application segmentation of logical messages and Putting and getting a group that spans units of work.

 

Putting and getting a group that spans units of work

In the previous case, messages or segments cannot start to leave the node (if its destination is remote) or start to be retrieved until all of the group has been put and the unit of work is committed. This may not be what you want if it takes a long time to put the whole group, or if queue space is limited on the node. To overcome this, the group can be put in several units of work.

If the group is put within multiple units of work, it is possible for some of the group to commit even when a failure of the putting application occurs. The application must therefore save status information, committed with each unit of work, which it can use after a restart to resume an incomplete group. The simplest place to record this information is in a STATUS queue. If a complete group has been successfully put, the STATUS queue is empty.

If segmentation is involved, the logic is similar. In this case, the StatusInfo must include the Offset.

Here is an example of putting the group in several units of work: 

   PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT
 
   /* First UOW */
 
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   StatusInfo = GroupId,MsgSeqNumber from MQMD
   MQPUT (StatusInfo to STATUS queue) PMO.Options = MQPMO_SYNCPOINT
   MQCMIT
 
   /* Next and subsequent UOWs */
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQGET (from STATUS queue) GMO.Options = MQGMO_SYNCPOINT
   StatusInfo = GroupId,MsgSeqNumber from MQMD
   MQPUT (StatusInfo to STATUS queue) PMO.Options = MQPMO_SYNCPOINT
   MQCMIT
 
   /* Last UOW */
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
   MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP
   MQGET (from STATUS queue) GMO.Options = MQGMO_SYNCPOINT
   MQCMIT

If all the units of work have been committed, the entire group has been put successfully, and the STATUS queue is empty. If not, the group must be resumed at the point indicated by the status information. MQPMO_LOGICAL_ORDER cannot be used for the first put, but can thereafter.

Restart processing looks like this: 

   MQGET (StatusInfo from STATUS queue) GMO.Options = MQGMO_SYNCPOINT
   if (Reason == MQRC_NO_MSG_AVAILABLE)
      /* Proceed to normal processing */
      ...
 
   else
      /* Group was terminated prematurely */
      Set GroupId, MsgSeqNumber in MQMD to values from Status message
      PMO.Options = MQPMO_SYNCPOINT
      MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
 
      /* Now normal processing is resumed.
         Assume this is not the last message */
      PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT
      MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
      MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP
      StatusInfo = GroupId,MsgSeqNumber from MQMD
      MQPUT (StatusInfo to STATUS queue) PMO.Options = MQPMO_SYNCPOINT
      MQCMIT

From the getting application, you may want to start processing the messages in a group before the whole group has arrived. This improves response times on the messages within the group, and also means that storage is not required for the entire group.

For recovery reasons, each message must be retrieved within a unit of work. However, in order to realize the above benefits, several units of work must be used for each group of messages.

As with the corresponding putting application, this requires status information to be recorded somewhere automatically as each unit of work is committed. Again, the simplest place to record this information is on a STATUS queue. If a complete group has been successfully processed, the STATUS queue is empty.

Note:
For intermediate units of work, you can avoid the MQGET calls from the STATUS queue by specifying that each MQPUT to the status queue is a segment of a message (that is, by setting the MQMF_SEGMENT flag), instead of putting a complete new message for each unit of work. In the last unit of work, a final segment is put to the status queue specifying MQMF_LAST_SEGMENT, and then the status information is cleared with an MQGET specifying MQGMO_COMPLETE_MSG.

During restart processing, instead of using a single MQGET to get a possible status message, browse the status queue with MQGMO_LOGICAL_ORDER until you reach the last segment (that is, until no further segments are returned). In the first unit of work after restart, also specify the offset explicitly when putting the status segment.

In the following example, we consider only messages within a group. It is assumed that the application's buffer is always large enough to hold the entire message, whether or not the message has been segmented. MQGMO_COMPLETE_MSG is therefore specified on each MQGET. The same principles apply if segmentation is involved (in this case, the StatusInfo must include the Offset).

For simplicity, we assume that a maximum of 4 messages should be retrieved within a single UOW: 

   msgs = 0    /* Counts messages retrieved within UOW */
   /* Should be no status message at this point */
 
   /* Retrieve remaining messages in the group */
   do while ( GroupStatus == MQGS_MSG_IN_GROUP )
 
      /* Process up to 4 messages in the group */
      GMO.Options = MQGMO_SYNCPOINT | MQGMO_WAIT
                  | MQGMO_LOGICAL_ORDER
      do while ( (GroupStatus == MQGS_MSG_IN_GROUP) && (msgs < 4) )
         MQGET
         msgs = msgs + 1
         /* Process this message */
         ...
      /* end while
 
      /* Have retrieved last message or 4 messages  */
      /* Update status message if not last in group */
      MQGET (from STATUS queue) GMO.Options = MQGMO_SYNCPOINT
      if ( GroupStatus == MQGS_MSG_IN_GROUP )
         StatusInfo = GroupId,MsgSeqNumber from MQMD
         MQPUT (StatusInfo to STATUS queue) PMO.Options = MQPMO_SYNCPOINT
      MQCMIT
      msgs = 0
   /* end while
 
   if ( msgs > 0 )
      /* Come here if there was only 1 message in the group */
      MQCMIT

If all of the units of work have been committed, then the entire group has been retrieved successfully, and the STATUS queue is empty. If not, then the group must be resumed at the point indicated by the status information. MQGMO_LOGICAL_ORDER cannot be used for the first retrieve, but can thereafter.

Restart processing looks like this: 

   MQGET (from STATUS queue) GMO.Options = MQGMO_SYNCPOINT
   if (Reason == MQRC_NO_MSG_AVAILABLE)
      /* Proceed to normal processing */
      ...
 
   else
      /* Group was terminated prematurely */
      /* The next message on the group must be retrieved by matching
         the sequence number and group id with those retrieved from the
         status information. */
      GMO.Options = MQGMO_COMPLETE_MSG | MQGMO_SYNCPOINT | MQGMO_WAIT
      MQGET GMO.MatchOptions = MQMO_MATCH_GROUP_ID | MQMO_MATCH_MSG_SEQ_NUMBER,
            MQMD.GroupId      = value from Status message,
            MQMD.MsgSeqNumber = value from Status message plus 1
      msgs = 1
      /* Process this message */
      ...
 
      /* Now normal processing is resumed */
      /* Retrieve remaining messages in the group */
      do while ( GroupStatus == MQGS_MSG_IN_GROUP )
 
         /* Process up to 4 messages in the group */
         GMO.Options = MQGMO_COMPLETE_MSG | MQGMO_SYNCPOINT | MQGMO_WAIT
                     | MQGMO_LOGICAL_ORDER
         do while ( (GroupStatus == MQGS_MSG_IN_GROUP) && (msgs < 4) )
            MQGET
            msgs = msgs + 1
            /* Process this message */
            ...
 
         /* Have retrieved last message or 4 messages  */
         /* Update status message if not last in group */
         MQGET (from STATUS queue) GMO.Options = MQGMO_SYNCPOINT
         if ( GroupStatus == MQGS_MSG_IN_GROUP )
            StatusInfo = GroupId,MsgSeqNumber from MQMD
            MQPUT (StatusInfo to STATUS queue) PMO.Options = MQPMO_SYNCPOINT
         MQCMIT
         msgs = 0

 

WebSphere is a trademark of the IBM Corporation in the United States, other countries, or both.

 

IBM is a trademark of the IBM Corporation in the United States, other countries, or both.