Injecting failures into a high availability system

We can inject failures into the system to check that the high availability behavior functions as you expect.

This facility is provided to support acceptance testing of a highly available configuration and should only be used for that purpose. Injecting a failure into the system will cause resources to be disabled or failed over from one server to another and will disrupt the workload.

We can send a JMX command to a messaging engine MBean to simulate a failure in the high availability system. Injecting failures provides a useful way to undertake advanced verification or preproduction testing. We should not inject a failure into a production system.

There are two types of messaging engine failure that we can simulate: local error and global error. For more information about error types, see Messaging engine recovery from exception conditions.

Tasks

1. Start the wsadmin client.

   (iSeries) Note: (iSeries) The wsadmin scripting client is run from Qshell. (iSeries) See Configure Qshell to run WebSphere scripts .

   For more information about the wsadmin client, see wsadmin scripting tool.

2. Use a JMX command to create a variable and set its value to the messaging engine, or engines, to fail.

   In Jython:

   mbean_name = AdminControl.queryNames("type=SIBMessagingEngine,name=messaging_engine_name,*" )
   

   In Jacl:

   set mbean_name [$AdminControl queryNames type=SIBMessagingEngine,name=messaging_engine_name,*]
   

3. Use a JMX command to inject the failure, using the variable we created in the previous step.

   To inject a local error in Jython:

   AdminControl.invoke(mbean_name, "injectFault", "LocalError")

   To inject a global error in Jython:

   AdminControl.invoke(mbean_name, "injectFault", "GlobalError")

   To inject a local error in Jacl:

   $AdminControl invoke $mbean_name injectFault LocalError

   To inject a global error in Jacl:

   $AdminControl invoke $mbean_name injectFault GlobalError
   

Use the administrative console to view the results. If we have configured the system for failover, a local error should cause the messaging engine to be failed over to another server. A global error does not cause a failover.

Example

For example, to inject a global error into a messaging engine named myNode01.server1-bus1, use the following commands:

In Jython:

myMBean = AdminControl.queryNames("type=SIBMessagingEngine,name=myNode01.server1-bus1,*")
$AdminControl invoke $myMBean injectFault GlobalError

In Jacl:

set myMBean [$AdminControl queryNames type=SIBMessagingEngine,name=myNode01.server1-bus1,*]
AdminControl.invoke(myMBean, "injectFault", "GlobalError")

Related:

Messaging engine recovery from exception conditions