Specify offload options for shared messages

We can choose where the message data for a shared queue message is stored, either in a Db2 table or a shared message data set (SMDS). You can also select which messages are offloaded, based on the size of the message and the current usage of the coupling facility structure (CF).

The message data for shared queues can be offloaded from the coupling facility and stored in either a Db2 table or in an IBM MQ managed data set called a shared message data set (SMDS).

For messages larger than the coupling facility entry size of 63 KB, offloading message data to a SMDS can have a significant performance improvement compared with offloading to Db2.

Every shared queue message is still managed using a list entry in a coupling facility structure, but when the message data is offloaded to the SMDS, the coupling facility entry only contains some control information and a list of references to the relevant disk blocks where the message is stored. Using this mechanism means the amount of coupling facility element storage required for each message is only a fraction of the actual size of the message.

    Selecting where the shared queue messages are stored

    The selection of SMDS or Db2 shared message storage is controlled with the OFFLOAD(SMDS|DB2) parameter on the CFSTRUCT definition. OFFLOAD(SMDS) is the default value.

    This parameter also requires the CFSTRUCT to use CFLEVEL(5) or greater. Only queue managers at IBM WebSphere MQ Version 7.1 or higher can connect to a CF structure at this level.

    It is only possible to alter a structure up to CFLEVEL(5) if all queue managers in the queue sharing group are at IBM WebSphere MQ Version 7.1 or higher.

    The OFFLOAD parameter is only valid from CFLEVEL(5). See DEFINE CFSTRUCT for more details.

    OFFLOAD(DB2) is supported primarily for migration purposes.

    Selecting which shared queue messages are offloaded

    Message data is offloaded to SMDS or Db2 based on the size of the message data, and the current usage of the coupling facility structure. There are three rules, and each rule specifies a matching pair of parameters. These parameters are a corresponding coupling facility structure usage threshold percentage ( OFFLDnTH ) and a message size limit ( OFFLDnSZ ).

    The current implementation of the three rules is specified using the following pairs of keywords:

    • OFFLD1TH and OFFLD1SZ
    • OFFLD2TH and OFFLD2SZ
    • OFFLD3TH and OFFLD3SZ

    Rule pair Default value Description
    Rule pair 1 OFFLD1TH(70) and OFFLD1SZ(32K) If the coupling facility structure is more than 70% full offload data for messages exceeding 32 KB
    Rule pair 2 OFFLD2TH(80) and OFFLD2SZ(4K) If the coupling facility structure is more than 80% full offload data for messages exceeding 4 KB
    Rule pair 3 OFFLD3TH(90) and OFFLD3SZ(0K) If the coupling facility structure is more than 90% full offload data for messages exceeding 0 KB (all messages)
    Important: Messages exceeding 63.75 KB are always offloaded. The value 64K for OFFLD x SZ is used to indicate that the rule is not in effect.

    Each message which is offloaded still requires 0.75 KB of storage in the coupling facility.

    The three offload rules which can be specified for each structure are intended to be used as follows.

    • Performance

      • When there is plenty of space in the application structure, message data should only be offloaded if it is too large to store in the structure, or if it exceeds some lower message size threshold such that the performance value of storing it in the structure is not worth the amount of structure space that it would need.
      • If a specific message size threshold is required, it is conventionally specified using the first offload rule.

    • Capacity

      • When there is very little space in the application structure, the maximum amount of message data should be offloaded so as to make the best use of the remaining space.
      • The third offload rule is conventionally used to indicate that when the structure is nearly full, most messages should be offloaded, so the entries in the application structure will be typically of the minimum size (requiring about 0.75K bytes).
      • The usage threshold parameter should be chosen based on the application structure size and the maximum anticipated backlog. For example, if the maximum anticipated backlog is 1M messages, then the amount of structure storage required for this number of messages is about 0.75G bytes. This means for example that if the structure is about 10G bytes, the usage threshold for offloading all messages must be set to 92% or lower.
      • Structure space is divided into elements and entries, and even though there may be enough space overall, one of these may run out before the other. The system provides AUTOALTER capabilities to adjust the ratio when necessary, but this is not very sensitive, so the amount of space actually available may be somewhat less. It may be better therefore to aim to use not more than 90% of the maximum structure space, so in the previous example, the usage threshold for offloading all messages would be better set around 80%.

    • Cushioned transition:

      • As the amount of space left in the coupling facility structure decreases, it would be undesirable to have a large sudden change in the performance characteristics. It is also undesirable for coupling facility management to have a sudden threshold change in the typical ratio of entries to elements being used.
      • The second offload rule is conventionally used to provide some intermediate cushion between the performance and capacity biased offload rules. It can be set to cause a significant increase in offload activity when the space used in the coupling facility structure exceeds an intermediate threshold. This means that the remaining space is used up more slowly, and gives the coupling facility automatic alter processing more time to adapt to the higher usage levels.

    If the coupling facility structure cannot be expanded, and there is a need to store at least some predetermined number of messages, the third rule can be modified as necessary to ensure that offloading of data for all messages starts at an appropriate threshold to ensure space is reserved for that predetermined number of messages.

    For example, if the coupling facility structure size is 4 GB, and the predetermined number of messages is 1 million, then 1,000,000 * 0.75 KB are needed, which is 768 MB, 18.75% of 4 GB. In this case the threshold for offloading all messages needs to be set around 80% rather than 90%. This gives parameters OFFLD3TH(80) and OFFLD3SZ(0K) . The other offload parameters would also need to be adjusted.

    If it is found that offloading very small messages has a significant performance impact, but the relative impact is less for larger messages, then the usage thresholds for the other rules can be reduced to offload larger messages earlier, leaving more space in the structure for small messages before they need to be offloaded.

    For example, if messages exceeding 32KB occur frequently but the elapsed time performance for offloading them (as determined from RMF statistics or application performance) is very similar to that for keeping them in the coupling facility, then the threshold for the first rule could be set to 0% to offload all such messages. This gives parameters OFFLD1TH(0) and OFFLD1SZ(32K). Again the other offload parameters would need to be adjusted.

    If there are many messages around specific intermediate sizes, such as 16 KB and 6 KB, then it might be useful to change the message size option for the second rule so that the larger ones get offloaded at a fairly low usage threshold, saving a significant amount of space, but the smaller ones still get stored only in the coupling facility.

Parent topic: Where are shared queue messages held?