Killing a cluster member during a request

Killing the Java virtual machine (JVM) that a cluster member is running on is the best way to see what will happen if a cluster member crashes during a request. When killing a cluster member before a request, the effect is the same as seen in Stopping a cluster member.

When you kill a cluster member during a request, sometimes that request is dispatched to another cluster member. There is a mechanism for the plug-in to swap over the request in the middle of the process. However, this mechanism does not guarantee high-availability of all requests. In such case, a cluster member that is functioning will be chosen for a re-issued request, which can then be tried again. The request will then be completed.

If you wish to see this, follow these steps.

1. Verify that all the cluster members are running within the cluster. Check this by following the steps described in Workload management with the plug-in.

Cycle through to make sure that all the cluster members are available. There is no need to repeat the changes to the workload management policy.

2. In the plugin-cfg.xml file, change the Log tag. Set the Log tag so that the line looks like this:

<Log LogLevel="Trace" Name="C:\WebSphere\AppServer\logs\killclustermember.log"/>

3. Save and close the plugin-cfg.xml file.

4. Find the Java process ID of the application server cluster member you want to kill. On AIX, we used the following command to find the process ID of our cluster member named PluginMember2:

ps -ef | grep PluginMember2

Note the process ID of your cluster member.

5. Open a browser and go to:

http://<yourWebServer>/wlm/beenthere

6. Click the Execute button until the Servlet Server Name field shows the name of the cluster member before the one you are going to kill. Monitor how the round robin process is cycling to find this out. The next time that you click Execute, the request needs to go to the cluster member that you intend to kill. This assumes that you are using appservers with equal weights or you need to make sure that none of your application servers reaches a weight of 0 while performing this test.

7. Set the Bean Iterations field to a large amount, for example 10000.

8. If you are running on AIX, now is a good time to use the tail command to monitor your plug-in log file.

9. Click the Execute button. This will start the request running on your cluster member. If you look at the plug-in log, you see that there is information passing between browser and appserver.

10. Kill the cluster member you located in step 4. The request will subsequently be rerouted to another cluster member. In this case in Example 5-20, the request is dispatched to another cluster member.

Example 5-20 Plug-in trace with request failover 

...
### At first, the request is processing by the PluginMember2
TRACE: ws_server: serverHasReachedMaxConnections: currentConnectionsCount 0, maxConnectionsCount -1.
TRACE: ws_server_group: lockedServerGroupUseServer: Server was2Node_PluginMember2 picked, weight 0.
TRACE: ws_server_group: serverGroupIncrementConnectionCount: Server was2Node_PluginMember2 picked, pendingConnectionCount 1 totalConnectionsCount 9.
TRACE: ws_common: websphereFindTransport: Finding the transport
...
### The plugin detects the error of the PluginMember2 and the request is dispatched to the PluginMember1 again
ERROR: ws_common: websphereExecute: Failed to read from a new stream; App Server may have gone down during read
TRACE: lib_stream: destroyStream: Destroying the stream
STATS: ws_server: serverSetFailoverStatus: Server was2Node_PluginMember2 : pendingConnections 0 failedConnections 2 affinityConnections 0 totalConnections 8.
ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'was2Node_PluginMember2'on host 'app2.itso.ibm.com'; will try another one
TRACE: ws_common: websphereHandleSessionAffinity: Checking for session affinity
TRACE: ws_common: websphereHandleSessionAffinity: Checking the SSL session id
TRACE: lib_htrequest: htrequestGetCookieValue: Looking for cookie: 'SSLJSESSION'
TRACE: lib_htrequest: htrequestGetCookieValue: No cookie found for: 'SSLJSESSION'
TRACE: ws_common: websphereHandleSessionAffinity: Checking the cookie affinity: JSESSIONID
TRACE: lib_htrequest: htrequestGetCookieValue: Looking for cookie: 'JSESSIONID'
TRACE: lib_htrequest: htrequestGetCookieValue: No cookie found for: 'JSESSIONID'
TRACE: ws_common: websphereHandleSessionAffinity: Checking the url rewrite affinity: jsessionid
TRACE: ws_common: websphereParseSessionID: Parsing session id from '/wlm/beenthere'
TRACE: ws_common: websphereParseSessionID: Failed to parse session id
TRACE: ws_server_group: serverGroupNextRoundRobinServer: Round Robin load balancing
TRACE: ws_server_group: serverGroupNextRoundRobinServer: numPrimaryServers is 2
TRACE: ws_server_group: assureWeightsValid: group PluginCluster
TRACE: ws_server_group: serverGroupGetFirstPrimaryServer: getting the first primary server
TRACE: ws_server_group: weights_need_reset: was1node_PluginMember1: 1 max, 0 cur.
TRACE: ws_server_group: serverGroupGetNextPrimaryServer: getting the next primary server
TRACE: ws_server_group: weights_need_reset: was2Node_PluginMember2: 1 max, 0 cur.
TRACE: ws_server_group: serverGroupGetNextPrimaryServer: getting the next primary server
TRACE: ws_server_group: weights_need_reset: Time to reset the weights
TRACE: ws_server_group: serverGroupGetFirstServer: getting the first server
TRACE: ws_server_group: serverGroupGetNextServer: getting the next server
TRACE: ws_server_group: serverGroupGetNextServer: getting the next server
TRACE: ws_server_group: assureWeightsValid: max multiplication factor 1.
TRACE: ws_server_group: serverGroupGetFirstServer: getting the first server
TRACE: ws_server_group: assureWeightsValid: Server was2Node_PluginMember2: 1 max, 1 cur.
TRACE: ws_server_group: serverGroupGetNextServer: getting the next server
TRACE: ws_server_group: assureWeightsValid: Server was1node_PluginMember1: 1 max, 1 cur.
TRACE: ws_server_group: serverGroupGetNextServer: getting the next server
TRACE: ws_server_group: serverGroupGetNextPrimaryServer: getting the next primary server
TRACE: ws_server_group: serverGroupGetFirstPrimaryServer: getting the first primary server
TRACE: ws_server: serverHasReachedMaxConnections: currentConnectionsCount 0, maxConnectionsCount -1.
STATS: ws_server_group: serverGroupCheckServerStatus: Checking status of was1node_PluginMember1, ignoreWeights 0, markedDown 0, retryNow 0, wlbAllows 1 reachedMaxConnectionsLimit 0
TRACE: ws_server: serverHasReachedMaxConnections: currentConnectionsCount 0, maxConnectionsCount -1.
TRACE: ws_server_group: lockedServerGroupUseServer: Server was1node_PluginMember1 picked, weight 0.
TRACE: ws_server_group: serverGroupIncrementConnectionCount: Server was1node_PluginMember1 picked, pendingConnectionCount 1 totalConnectionsCount 26.
TRACE: ws_common: websphereFindTransport: Finding the transport
TRACE: ws_common: websphereFindTransport: Setting the transport(case 2): app1.itso.ibm.com on port 9088

...

  Prev | Home | Next

 

WebSphere is a trademark of the IBM Corporation in the United States, other countries, or both.

 

IBM is a trademark of the IBM Corporation in the United States, other countries, or both.