Injecting failures into a high availability system

We can inject failures into the system to check that the high availability behavior functions as you expect.

This facility is provided to support acceptance testing of a highly available configuration and should only be used for that purpose. Injecting a failure into the system will cause resources to be disabled or failed over from one server to another and will disrupt the workload.

We can send a JMX command to a messaging engine MBean to simulate a failure in the high availability system. Injecting failures provides a useful way to undertake advanced verification or preproduction testing. You should not inject a failure into a production system. There are two types of messaging engine failure that we can simulate: local error and global error. For more information about error types, see Messaging engine recovery from exception conditions.

 

1. Start the wsadmin client.

   See about the wsadmin client, see Use the wsadmin scripting tool.

2. Use a JMX command to create a variable and set its value to the messaging engine, or engines, to fail. In Jython:

   mbean_name = AdminControl.queryNames("type=SIBMessagingEngine,name=messaging_engine_name,*" )
   

   In Jacl:

   set mbean_name [$AdminControl queryNames type=SIBMessagingEngine,name=messaging_engine_name,*]
   

3. Use a JMX command to inject the failure, by using the variable you created in the previous step. To inject a local error in Jython:

   AdminControl.invoke(mbean_name, "injectFault", "LocalError")
   

   To inject a global error in Jython:

   AdminControl.invoke(mbean_name, "injectFault", "GlobalError")
   

   To inject a local error in Jacl:

   $AdminControl invoke $mbean_name injectFault LocalError
   

   To inject a global error in Jacl:

   $AdminControl invoke $mbean_name injectFault GlobalError
   

 

Results

Use the admin console to view the results. If we have configured the system for failover, a local error should cause the messaging engine to be failed over to another server. A global error does not cause a failover.

 

Example

For example, to inject a global error into a messaging engine named myNode01.server1-bus1, use the following commands:In Jython:

myMBean = AdminControl.queryNames("type=SIBMessagingEngine,name=myNode01.
server1-bus1,*")

$AdminControl invoke $myMBean injectFault GlobalError

In Jacl:

set myMBean [$AdminControl queryNames type=SIBMessagingEngine,name=myNode01.
server1-bus1,*]

AdminControl.invoke(myMBean, "injectFault", "GlobalError")