Managing log files

Allocate sufficient space for your log files. For linear logging, we can delete old log files when they are no longer required.


Information specific to circular logging

If you are using circular logging, ensure that there is sufficient space to hold the log files when you configure your system (see Log defaults for IBM MQ and Queue manager logs). The amount of disk space used by the log does not increase beyond the configured size, including space for secondary files to be created when required.


Information specific to linear logging

If you are using a linear log, the log files are added continually as data is logged, and the amount of disk space used increases with time. If the rate of data being logged is high, disk space is used rapidly by new log files.

Over time, the older log files for a linear log are no longer required to restart the queue manager or to perform media recovery of any damaged objects. The following methods determine which log files are still required:

    Logger event messages
    When a significant event occurs, for example a record media image, logger event messages are generated. The contents of logger event messages specify the log files that are still required for queue manager restart, and media recovery. For more information about logger event messages, see Logger events

    Queue manager status
    Running the MQSC command, DISPLAY QMSTATUS, or the PCF command, Inquire Queue Manager Status, returns queue manager information, including details of the required log files. For more information about MQSC commands, see Script (MQSC) Commands, and for information about PCF commands, see Automate administration tasks.

    Queue manager messages
    Periodically, the queue manager issues a pair of messages to indicate which of the log files are needed:

    • Message AMQ7467 gives the name of the oldest log file required to restart the queue manager. This log file and all newer log files must be available during queue manager restart.
    • Message AMQ7468 gives the name of the oldest log file needed for media recovery.

To determine "older" and "newer" log files, use the log file number rather than the modification times applied by the file system.


Information applicable to both types of logging

Only log files required for queue manager restart, active log files, are required to be online. Inactive log files can be copied to an archive medium such as tape for disaster recovery, and removed from the log directory. Inactive log files that are not required for media recovery can be considered as superfluous log files. We can delete superfluous log files if they are no longer of interest to your operation.

If any log file that is needed cannot be found, operator message AMQ6767 is issued. Make the log file, and all subsequent log files, available to the queue manager and try the operation again.

Cleaning log extents automatically - linear logging only

From IBM MQ Version 9.0.2 we have the option to use automatic management of linear log extents no longer required for recovery.

We use the LogManagement attribute in the Log stanza of the qm.ini file, or by using the IBM MQ Explorer, to set up automatic management. See Queue manager logs for more information.

See the LOG parameter of DISPLAY QMSTATUS for more details about the operation of the log, and the following commands for using the log:

Taking media images automatically - linear logging only

From IBM MQ Version 9.0.2 there is an overall switch to control whether the queue manager automatically writes media images, the default being that the switch has not been set.

We can control whether automatic media imaging occurs, and the frequency of the process, by using the following queue manager attributes:

    IMGSCHED
    Whether the queue manager writes media images automatically

    IMGINTVL
    Frequency for writing media images, in minutes

    IMGLOGLN
    Megabytes of log written since previous media image of an object.

If we have a critical time during the day when workload is very heavy and you want to be sure that system throughput is not impacted by taking automatic media images, you might wish to temporarily switch off automatic media imaging by setting IMGSCHED(MANUAL).

We can switch IMGSCHED at any time during the workload.Attention: MEDIALOG will not be moved forward if you are not taking media images, so you need to, either be archiving extents, or be sure we have sufficient disk space. We can also control automatic and manual media images for other user-defined objects:

  • Authentication information
  • Channel
  • Client connection
  • Listener
  • Namelist
  • Process
  • Alias queue
  • Local queue
  • Service
  • Topic

For internal system objects, such as the object catalog and the queue manager object, the queue manager automatically writes media images as appropriate.

See ALTER QMGR for more information on the attributes.

We can also enable or disable automatic and manual media images for local and permanent dynamic queues only. You do this using the IMGRCOVQ queue attribute.

See ALTER QUEUES for more information on the IMGRCOVQ attribute.

Notes:
  1. Media images are supported only if you are using linear logging. If we have enabled automatic media images, but are using circular logging, an error message is issued and the automatic media images attribute of the queue manager is disabled.
  2. If we have enabled automatic media images, but have not specified a frequency, either minutes or megabytes of log, an error message is issued and no automatic media images are written.
  3. We can manually record a media image using rcdmqimg when we have set IMGSCHED(AUTO), if you want.

    This enables you to take media images at a time that is suitable for your enterprise, for example, when your system is quiet. Automatic media imaging takes account of these manual media images, because taking a manual media image resets the interval and log length, before which the next automatic media image is taken.

  4. In IBM MQ Version 9.0.2, the queue manager writes persistent messages only in media images, not nonpersistent messages. This can reduce the size of media images when migrating to IBM MQ Version 9.0.2 or later

Deciding how to set IMGLOGLN and IMGINTVL

Make IMGLOGLN and IMGINTVL large enough, so the queue manager is only spending a fraction of its time recording media images, but small enough so that:

  • Damaged objects can be recovered in a reasonable amount of time, and
  • Small enough so that your log fits on your disk without running out of space.

If you set IMGLOGLN, a good practice is to make IMGLOGLN many times the amount of data on your queues and many times the data rate of your workload. The larger you make IMGLOGLN, the less time your queue manager spends recording media images.

Similarly, if you set IMGINTVL, a good practice is to make IMGINTVL many times the amount of time the queue manager takes to record a media image. We can find out how long it takes to record a media image by recording one manually.

If you make IMGLOGLN and IMGINTVL too large, recovering a damaged object might take a very long time, because all the extents since the last media image have to be replayed.

Make IMGLOGLN and IMGINTVL small enough, so that the maximum time taken to recover a damaged object is acceptable to you.

Making IMGLOGLN and IMGINTVL very large, means that the log grows very large because media images are recorded so rarely. Attention: Ensure that a log of this size fits comfortably on your log filesystem, as your workload will get backed out if the log filesystem fills completely.

We can set both IMGINTVL and IMGLOGLN. This might be useful to ensure that automatic media images are taken regularly during heavy workload (controlled by IMGLOGLN), but are still taken occasionally when the workload is very light (controlled by IMGINTVL).

IMGINTVL and IMGLOGLN are targets for the interval and log data length between which automatic media images are taken.

These attributes should not be seen as a fixed maximum or minimum. In fact, the queue manager might decide to schedule an automatic media image sooner, if the queue manager perceives that it is a really good time:

  • Because the queue is empty, so taking the media image is the most efficient in terms of performance, and
  • A media image has not been recorded for a while

On occasion the gap between automatic media images might be somewhat longer than either, or both, of IMGINTVL and IMGLOGLN.

The gap between media images might be larger than IMGLOGLN if the amount of data on queues is approaching IMGLOGLN. The gap between media images might be larger than IMGINTVL if it takes almost as long as IMGINTVL to record a media image.

This is poor practice because the queue manager would be spending much of its time recording media images.

When using automatic media image recording, the queue manager records a media image for each object and queue individually, so the queue manager tracks the interval and log length between images separately for each object.

Gradually over time, recording of media images becomes staggered, instead of recording media images for all objects at the same time. This staggering spreads out the performance impact of recording media images, and is another advantage of using automatic recording of media images over manual recording.

Taking media images manually - linear logging only

Recording a media image of a queue involves writing all the persistent messages from that queue to the log. For queues containing large volumes of message data, this involves writing a large amount of data to the log, and this process can impact the performance of the system while it is happening.

Recording media images of other objects is likely to be comparatively quick, since the media image of other objects does not contain user data.

You need to carefully consider when to record the media images of queues, so that the process does not interfere with your peak workload.

You must record the media image of all objects regularly, in order to update the oldest log extent needed for media recovery.

A good time to record the media image of a queue is when it is empty, because at that point no message data is written to the log. Conversely, a bad time is when the queue is very deep or has very large messages on it.

A good time to record the media image of a queue is when your system is quiet; whereas a bad time is during peak workload. If your workload is always quiet at midnight, for example, you might decide to record media images at midnight every night.

Staggering the recording of each of your queues can spread the performance impact out, and so lessen its effect. The longer it has been since you last recorded media images, the more important it becomes to record them, as the number of log extents required for media recovery is increasing.

Note: When performing media recovery, all the required log files must be available in the log file directory at the same time. Make sure that you take regular media images of any objects you might want to recover to avoid running out of disk space to hold all the required log files. For example, to take a media image of all your objects in your queue manager, run the rcdmqimg command as shown in the following examples:

    On Windows 
    rcdmqimg -m QMNAME -t all *

    On UNIX and Linux 
    rcdmqimg -m QMNAME -t all "*"

Running rcdmqimg moves the media log sequence number (LSN) forwards. For further details on log sequence numbers, see Dumping the contents of the log using the dmpmqlog command. rcdmqimg does not run automatically, therefore must be run manually or from an automatic task we have created. For more information about this command, see rcdmqimg and dmpmqlog. Note: Messages AMQ7467 and AMQ7468 can also be issued at the time of running the rcdmqimg command.

Partial media images

It is good practice to use IBM MQ messages only for data that is expected to be consumed in the near future, so that each message is on a queue for a relatively short amount of time.

Conversely, it is poor practice to use IBM MQ messages to store data long term like a database.

It is also good practice to ensure your queues are relatively shallow, and poor practice to have deep queues whose messages have been on the queue for a long time.

By following these guidelines, you enables the queue manager to optimize the performance of automatic recording of media images.

Recording the media image of an empty queue is very efficient (from a performance point of view) whereas taking the media image of a queue with a large amount of data on it is very inefficient, because all that data has to be written to the log in the media image.

For shallow queues with recently put messages on it, the queue manager can make a further optimization.

If all the messages currently on the queue were put in the recent past, the queue manager might be able to record the media image on behalf of a time (recovery point) just before all the messages were put, and so be able to record the image of the empty queue. This process is very low cost in terms of performance.

If all the messages that were on the queue at the recovery point have subsequently been got, these messages do not need to be recorded in the media image, as they are no longer on the queue.

This is called a partial media image. Then, in the unlikely event that the queue needs to be recovered, all logs records that relate to this queue since the last media image, will be replayed, so restoring all the recently put messages.

Even if there were a few messages on the queue at the recovery point, that are currently on the queue (and so have to be recorded in the partial media image) it is still more efficient to record this smaller partial media image, than a full media image of all messages.

Ensuring that messages remain on queues for a brief amount of time is likely to improve the performance of automatic recording of media images.