Application balancing trouble shooting

A list of symptoms and solutions associated with application balancing, using the DISPLAY APSTATUS command.

DIS APSTATUS(X) TYPE(APPL)

Symptom

The expected application is not listed.

Solution

• Verify the APPLTAG field is set correctly, either in code, or when the application is started.
• Investigate other listed applications in DIS APSTATUS(*) output to see if any are unexpected due to the name being formed incorrectly, or defaulting.
• Try running the command DIS APSTATUS(X) TYPE(LOCAL) where(MOVABLE eq NO) on each queue manager in the uniform cluster, to look for application instances which can not be distributed around the uniform cluster.

Symptom

The expected total number of applications are not listed.

Solution

• Verify we are actually launching the expected number of instances to connect to the uniform cluster
• Verify that the uniform cluster is communicating correctly and all queue managers are reporting application counts in DIS APSTATUS(X) TYPE(QMGR).

Symptom

The expected total number of applications are listed but some applications are flagged as not movable.

Solution

On each queue manager in the uniform cluster, use DIS APSTATUS(X) TYPE(LOCAL) where(MOVABLE eq NO) and investigate the IMMREASN field.

Symptom

The balanced state is UNKNOWN

Solution

This is a temporary state, and will resolve itself shortly. Retry the command in a while.

Symptom

The balanced state is NOTAPPLIC.

Solution

• If this queue manager is not in a uniform cluster, the balance state is always NOTAPPLIC as nothing can be rebalanced.
• In a uniform cluster, this means there has never been an application with this name connecting as movable. Information on this application is not distributed around the cluster.

  Use DIS APSTATUS(X) TYPE(LOCAL) where(MOVABLE eq NO) and investigate the IMMREASN field.

Symptom

The balanced state is NO

Solution

• Monitor this output across a period of time. If applications constantly connect and disconnect this might be the appropriate answer as the instances are not given chance to rebalance.
• Use DIS APSTATUS(X) TYPE(QMGR) to investigate the numbers on each queue manager, which indicates queue managers with a surplus, or deficit, number of instances and continue the investigation on those queue managers.

DIS APSTATUS(X) TYPE(QMGR)

Symptom

Not all queue managers in the uniform cluster are listed.

Solution

• Verify the BALSTATE is not NOTAPPLIC as that prevents information being flown around the uniform cluster.

  Use DIS APSTATUS(X) TYPE(LOCAL) to look at the IMMREASN field.

• Verify any missing queue managers are running.
• Verify the state of clustering, and that channels are running between this queue manager and the missing queue manager.

Symptom

A queue manager is listed as ACTIVE(NO)

Solution

• Verify any missing queue managers are running
• Verify the state of clustering and that channels are running between this queue manager and the inactive queue manager

Symptom

A queue manager has some immovable instances of an application.

Solution

On that queue manager in the uniform cluster, Use DIS APSTATUS(X) TYPE(LOCAL) where(MOVABLE eq NO) and investigate the IMMREASN field.

Symptom

The BALSTATE is unexpected.

Solution

• Monitor this over time, as the BALSTATE is the state when the queue manager last attempted to rebalance applications, which only happens periodically
• Are applications continually connecting and disconnecting? If so, this might prevent the application ever being rebalanced into a stable state.
• If BALSTATE stays unbalanced, look at the error logs on the queue managers that are BALSTATE(HIGH) and BALSTATE(LOW), which should indicate whether they are requesting application instances, and how many were permitted to be moved.
• Verify DIS APSTATUS(X) TYPE(LOCAL) where(IMMCOUNT gt 1) to see if there are instances which are failing to move when requested.

DIS APSTATUS(X) TYPE(LOCAL)

Symptom

An application instance is flagged as MOVABLE(NO)

Solution

• Is the IMMREASN field NOTCLIENT. If so, the application is using server bindings and therefore cannot be moved to another queue manager
• Is the IMMREASN field NOTRECONN. If so, the application is not connecting as a reconnectable client, and therefore cannot be moved to another queue manager.

  Use DIS CONN(*) TYPE(CONN) WHERE(CONNTAG eq 'xxx') CONNOPTS, where xxx is the CONNTAG from the DIS APSTATUS output, to see how they connect.

• Is the IMMREASN field APPNAMECHG. If so, the application instance is making multiple conversations on the same connection but changing the application name, which prevents a particular application instance from being moved.
• Is the IMMREASN field MOVING. If so, wait a while and the problem should disappear as the application instance has been requested to move.
• Otherwise, check the IMMDATE and IMMTIME fields to see if the application is only temporarily marked as immovable

Parent topic: Queue manager clusters troubleshooting