Configure the hang detection policy

Configure the hang detection policy

The hang detection option for WebSphere Application Server is turned on by default. We can configure a hang detection policy to accommodate the applications and environment so that potential hangs can be reported, providing earlier detection of failing servers. When a hung thread is detected, WAS notifies you so that we can troubleshoot the problem.

A common error in Java EE applications is a hung thread. A hung thread can result from a simple software defect (such as an infinite loop) or a more complex cause (for example, a resource deadlock). System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless a user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.

Using the hang detection policy, we can specify a time that is too long for a unit of work to complete. The thread monitor checks all managed threads in the system (for example, web container threads and object request broker (ORB) threads). Unmanaged threads, which are threads created by applications, are not monitored. For more information read Hung threads in Java Platform, Enterprise Edition applications.

The thread hang detection option is enabled by default.

Adjust or disable the hang detection policy

From the administrative console...
Servers > Application Servers server > Server Infrastructure > Administration > Custom Properties > New

Add the following properties:

Name Value Default
com.ibm.websphere.threadmonitor.interval The frequency, in seconds, at which managed threads in the selected application server will be interrogated. 180 seconds (three minutes)
com.ibm.websphere.threadmonitor.threshold The length of time, in seconds, in which a thread can be active before it is considered hung. Any thread that is detected as active for longer than this length of time is reported as hung. 600 seconds (ten minutes)
com.ibm.websphere.threadmonitor.false.alarm.threshold The number of times (T) that false alarms can occur before automatically increasing the threshold. It is possible that a thread that is reported as hung eventually completes its work, resulting in a false alarm. A large number of these events indicates that the threshhold value is too small. The hang detection facility can automatically respond to this situation: For every T false alarms, the threshold T is increased by a factor of 1.5. Set the value to zero (or less) to disable the automatic adjustment. 100
com.ibm.websphere.threadmonitor.dump.java This property calls the dumpThreads function. Set to true to execute the dumpThreads function when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore dump can be analyzed to determine what the reported thread and other related threads are doing.

Set to an integer value in the range 1 through Integer.MAX_VALUE to cause the dumpThreads function to be executed when a hung thread is detected and a WSVR0605W message is printed. The integer value indicates the maximum number of times dumpThreads will be executed.
To turn off WSVR0605W messages, set com.ibm.websphere.threadmonitor.interval=0.
By default, the dumpThreads function creates a javacore dump. See the topic Dumping threads in server processes using scripting for information about how to disable the creation of this dump. See the Java Diagnostics guide for more information about the contents and implications of this dump.

(ZOS) By default, the dumpThreads function creates a javacore dump, a Heapdump, and a TDUMP.
false (0)
com.ibm.websphere.threadmonitor.dump.java.track Number of intervals the monitor will call the dumpThreads function while a thread remains hung.

Set to an integer value in the range 2 through the value of com.ibm.websphere.threadmonitor.dump.java, which causes the dumpThreads function to be executed over subsequent monitor intervals in which a thread remains hung. The integer value indicates the maximum number of times dumpThreads is executed to track a hung thread. For example, a value of 2 causes the thread monitor to execute dumpThreads twice at the most for a hung thread, and once when the thread is initially detected as hung. The thread monitor can execute dumpThreadsand once again if the thread remains hung during the next thread monitor interval.

This property can only be used in conjunction with the com.ibm.websphere.threadmonitor.dump.java property.
(0) intervals
Optional: To monitor the activity of threads on which system alarms execute, add the following JVM generic arguments to the server settings.
-Dcom.ibm.websphere.alarmthreadmonitor.generate.javacore Set to any value to cause a javacore dump to be created when an hung system alarm thread is detected. The threads section of the javacore dump can be analyzed to determine what the reported thread and other related threads are doing. Unset
com.ibm.websphere.alarmthreadmonitor.checkinterval.millis The frequency, in milliseconds, at which system alarm threads are interrogated. Set the value to zero to disable system alarm hung thread detection. The maximum interval is 600000
(10 minutes). 10000
(10 seconds)
-Dcom.ibm.websphere.alarmthreadmonitor.threshold.millis Set to any value integer between 10000 and 600000 (10 minutes). This argument specifies the length of time, in milliseconds, that a system alarm thread can be active before it is considered non-responsive. Any system alarm thread that is detected as inactive for longer than this length of time is reported as hung. 10000
(10 seconds)

To add these arguments to the server settings:

Under Server Infrastructure on the server settings page in the administrative console, click Java and process management > Process definition.
Select Control.
Select Java virtual machine.
Add the arguments to the JVM generic arguments section.

Click Apply.
Click OK.
Save the changes. Make sure a file synchronization is performed before restarting the servers.
Restart the Application Server for the changes to take effect.

Subtopics

Hung threads in Java Platform, Enterprise Edition applications

Example: Adjusting the thread monitor to affect server hang detection

Related:

Dumping threads in server processes using scripting

Name	Value	Default
com.ibm.websphere.threadmonitor.interval	The frequency, in seconds, at which managed threads in the selected application server will be interrogated.	180 seconds (three minutes)
com.ibm.websphere.threadmonitor.threshold	The length of time, in seconds, in which a thread can be active before it is considered hung. Any thread that is detected as active for longer than this length of time is reported as hung.	600 seconds (ten minutes)
com.ibm.websphere.threadmonitor.false.alarm.threshold	The number of times (T) that false alarms can occur before automatically increasing the threshold. It is possible that a thread that is reported as hung eventually completes its work, resulting in a false alarm. A large number of these events indicates that the threshhold value is too small. The hang detection facility can automatically respond to this situation: For every T false alarms, the threshold T is increased by a factor of 1.5. Set the value to zero (or less) to disable the automatic adjustment.	100
com.ibm.websphere.threadmonitor.dump.java	This property calls the dumpThreads function. Set to true to execute the dumpThreads function when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore dump can be analyzed to determine what the reported thread and other related threads are doing. Set to an integer value in the range 1 through Integer.MAX_VALUE to cause the dumpThreads function to be executed when a hung thread is detected and a WSVR0605W message is printed. The integer value indicates the maximum number of times dumpThreads will be executed. To turn off WSVR0605W messages, set com.ibm.websphere.threadmonitor.interval=0. By default, the dumpThreads function creates a javacore dump. See the topic Dumping threads in server processes using scripting for information about how to disable the creation of this dump. See the Java Diagnostics guide for more information about the contents and implications of this dump. (ZOS) By default, the dumpThreads function creates a javacore dump, a Heapdump, and a TDUMP.	false (0)
com.ibm.websphere.threadmonitor.dump.java.track	Number of intervals the monitor will call the dumpThreads function while a thread remains hung. Set to an integer value in the range 2 through the value of com.ibm.websphere.threadmonitor.dump.java, which causes the dumpThreads function to be executed over subsequent monitor intervals in which a thread remains hung. The integer value indicates the maximum number of times dumpThreads is executed to track a hung thread. For example, a value of 2 causes the thread monitor to execute dumpThreads twice at the most for a hung thread, and once when the thread is initially detected as hung. The thread monitor can execute dumpThreadsand once again if the thread remains hung during the next thread monitor interval. This property can only be used in conjunction with the com.ibm.websphere.threadmonitor.dump.java property.	(0) intervals Optional: To monitor the activity of threads on which system alarms execute, add the following JVM generic arguments to the server settings.
-Dcom.ibm.websphere.alarmthreadmonitor.generate.javacore	Set to any value to cause a javacore dump to be created when an hung system alarm thread is detected. The threads section of the javacore dump can be analyzed to determine what the reported thread and other related threads are doing.	Unset
com.ibm.websphere.alarmthreadmonitor.checkinterval.millis	The frequency, in milliseconds, at which system alarm threads are interrogated. Set the value to zero to disable system alarm hung thread detection. The maximum interval is 600000 (10 minutes).	10000 (10 seconds)
-Dcom.ibm.websphere.alarmthreadmonitor.threshold.millis	Set to any value integer between 10000 and 600000 (10 minutes). This argument specifies the length of time, in milliseconds, that a system alarm thread can be active before it is considered non-responsive. Any system alarm thread that is detected as inactive for longer than this length of time is reported as hung.	10000 (10 seconds)