+

Search Tips   |   Advanced Search

Workload management component troubleshooting tips

If the workload management component is not properly distributing the workload across servers in multi-node configuration, use the following options to isolate the problem.


Eliminate environment or configuration issues

Determine if the servers are capable of serving the applications for which they have been enabled. Identify the cluster that has the problem.

If we are experiencing workload management problems related to HTTP requests, such as HTTP requests not being served by all members of the cluster, be aware that the HTTP plug-in balances the load across all servers defined in the PrimaryServers list if affinity has not been established. If we do not have a PrimaryServers list defined then the plug-in load balances across all servers defined in the cluster if affinity has not been established. If affinity has been established, the plug-in should go directly to that server for all requests.

For workload management problems relating to enterprise bean requests, such as enterprise bean requests not getting served by all members of a cluster:

The remainder of this article deals with enterprise bean workload balancing only. For more help on diagnosing problems in distributing web (HTTP) requests, view the "Web server plug-in troubleshooting tips" and "Web resource does not display" topics.


(iSeries) (Dist) Browse log files for WLM errors and CORBA minor codes

If we still encounter problems with enterprise bean workload management, the next step is to check the activity log for entries that show:


(iSeries) (Dist) To do this, use the Log and Trace Analyzer to open the service log (activity.log) on the affected servers, and look for the following entries:

(ZOS) To do this, open the service log on the affected servers, and look for the following entries:

If any of these warning are encountered, follow the user response given in the log. If, after following the user response, the warnings persist, look at any other errors and warnings in the Log and Trace Analyzer on the affected servers to look for:

You may also see exceptions with "CORBA" as part of the exception name, since WLM uses CORBA (Common Object Request Broker Architecture) to communicate between processes. Look for a statement in the exception stack specifying a "minor code". These codes denote the specific reason a CORBA call or response could not complete. WLM minor codes fall in range of 0x4921040 - 0x492104F. For an explanation of minor codes related to WLM, see the topic "Reference: Generated API documentation" for the package and class com.ibm.websphere.wlm.WsCorbaMinorCodes.


(iSeries) (Dist) Analyze PMI data

The purpose for analyzing the PMI data is to understand the workload arriving for each member of a cluster. The data for any one member of the cluster is only useful within the context of the data of all the members of the cluster.

Use the Tivoli Performance Viewer to verify that, based on the weights assigned to the cluster members (the steady-state weights), each server is getting the correct proportion of the requests.

To use the Tivoli Performance Viewer to capture PMI metrics, in the Tivoli Performance Viewer product navigation complete the following actions:

  1. Select Data Collection in the tree view. Servers that do not have PMI enabled are grayed out.

  2. For each server that data you wish to collect data on, click Specify...

  3. We can now enable the metrics. Set the monitoring level to low on the Performance Monitoring Setting panel

  4. Click OK

  5. We must hit Apply for the changes we have made to be saved.

WLM PMI metrics can be viewed on a server by server basis. In the Tivoli Performance Viewer select Node > Server > WorkloadManagement > Server/Client. By default the data is shown in raw form in a table, collected every 10 seconds, as an aggregate number. We can also choose to see the data as a delta or rate, add or remove columns, clear the buffer, reset the metrics to zero, and change the collection rate and buffer size.

After we have obtained the PMI data, we should calculate the percentage of numIncomingRequests for each member of the cluster to the total of the numIncomingRequests of all members of the cluster. A comparison of this percentage value to the percentage of weights directed to each member of the cluster provides an initial look at the balance of the workload directed to each member of a cluster.

In addition to the numIncomingRequests two other metrics show how work is balanced between the members of a cluster, numincomingStrongAffinityRequests and numIncomingNonWLMObjectRequests. These two metrics show the number of requests directed to a specific member of a cluster that could only be serviced by that member.

For example, consider a 3-server cluster. The following weights are assigned to each of these three servers:

Allow our cluster of servers to start servicing requests, and wait for the system to reach a steady state, that is the number of incoming requests to the cluster equals the number of responses from the servers. In such a situation, we would expect that the percentage of requests routed to each server to be:

Now let us consider a case where there are no incoming requests with neither strong affinity nor any non-WLM object requests.

In this scenario, let us assume that the PMI metrics gathered show the number of incoming requests for each server are:

Thus, the total number of requests coming into the cluster is: numIncomingRequestsCluster = numIncomingRequestsServer1 + numIncomingRequestsServer2 + numIncomingRequestsServer3 = 784

numincomingStrongAffinityRequests = 0

numIncomingNonWLMObjectRequests = 0

Can we decide based on this data if WLM is properly balancing the incoming requests among the servers in our cluster? Since there are no requests with strong affinity, the question we need to answer is, are the requests in the ratios we expect based on the assigned weights? The computation to answer that question is straightforward:

So WLM is behaving as designed, as the data are completely what is expected, based on the weights assigned the servers.

Now let us consider a 3-server cluster. We have assigned the following weights to each of these three servers:

Allow this cluster of servers to start servicing requests and wait for the system to reach a steady state, that is the number of incoming requests to the cluster equals the number of responses from the servers. In such a situation, we would expect that the percentage of requests that are routed to Server1-3 would be:

In this scenario, let us assume that the PMI metrics gathered show the number of incoming requests for each server are:

Thus, the total number of requests coming into the cluster:

In this case, we see that the number of requests was not evenly split among the three servers, as expected. Instead, the distribution is:

However, the correct interpretation of this data is the routing of requests is not perfectly balanced because Server1 had several hundred strong affinity requests. WLM attempts to compensate for strong affinity requests directed to 1 or more servers by distributing new incoming requests preferentially to servers that are not participating in transactional affinity, to compensate for those servers that are participating in transactions. In the case of incoming requests with strong affinity and non-WLM object requests, the analysis would be analogous to this case.

If, after we have analyzed the PMI data and accounted for transactional affinity and non-WLM object requests, the percentage of actual incoming requests to servers in a cluster to do not reflect the assigned weights, this indicates that requests are not being properly balanced. If this is the case, IBM recommends that you repeat the steps for eliminating environment and configuration issues and browsing log files before proceeding.


Resolve problem or contact IBM support

(iSeries) (Dist) If the PMI data or client logs indicate an error in WLM, collect the following information and contact IBM support.

If the client logs indicate an error in WLM, collect the following information and contact IBM support.

Check to see if the problem has been identified and documented using the links in the "Diagnose and fix problems: Resources for learning" topic. If we do not see a problem that resembles ours, or if the information provided does not solve our problem, contact IBM support for further assistance.

If we do not find our problem listed there, contact IBM Support.

(Dist) For current information available from IBM Support on known problems and their resolution, see the IBM Support page. We should also refer to this page before opening a PMR because it contains documents that can save you time gathering information needed to resolve a problem.


Related:

  • JDBC providers
  • Performance Monitoring Infrastructure (PMI)
  • Troubleshoot administration
  • Manage application servers
  • Modify the default web container configuration
  • Configure first failure data capture log file purges
  • Data access problems
  • Web server plug-in troubleshooting tips
  • Web resource is not displayed
  • View Data with the Tivoli Performance Viewer
  • High Performance Extensible Logging