Configure the memory leak policy

Configure the memory leak policy

The leak detection policy for the WAS is turned off by default. We can configure a leak detection, prevention, and action policy. When a classloader memory leak is detected, WAS writes informational messages to the log, and takes JVM heapdumps for troubleshooting the problem. Optionally, we can choose to have WAS mitigate, and if possible, fix the memory leak using reflection and other techniques.
System resources, such as CPU time due to garbage collection and the Java heap are consumed when a leak is present. A system can become unresponsive even though all other resources are available. Unless a protection and early warning system is built in, the system might remain in this degraded state and ultimately die due to an out of memory error.
We can use JVM custom properties to adjust the leak policy values, such as, enable and disable leak detection, action, and prevention. These custom properties are only applicable to a stand-alone server or managed application server and not to a node agent, admin agent, job manager, or deployment manager. When the application or the server is shutting down, WAS determines the classloaders that have leaked and are held references to all of associated loaded classes and objects. If a classloader leak is detected, a heapdump or systemdump is taken.
All persistent configurations of this service are completed using JVM custom properties. There are no administrative console panels. At runtime, use the MemoryLeakConfig and MemoryLeakAdmin mbeans for configuration and administration respectively; however, the configuration changes are not persisted until JVM custom properties are configured.
Configure the JVM com.ibm.ws.runtime.component.disableMemoryLeakService custom property to permanently disable the service.
Create or modify JVM custom properties to enable various aspects of the memory leak service, based on the table that follows.

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.detectAppCLLeaks
Description When the server is shutting down or an application stops, WAS determines the classloaders that have leaked and issues warnings and other additional information that aids in debugging the memory leak. See also the Improved classloader leak detection PMR.
Default false

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks
Description Enable proactive classloader leak mediation and fixing. When true, WebSphere Application Server mediates on behalf of the application to remedy any classloader leaks that are detected.
Default false

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.preventJreMemoryLeaks
Description Enable WebSphere Application Server to eliminate certain classes of memory leaks that are caused by the JRE loading singletons on the thread context classloader.
Default true

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.generateHeapDumps
Description Set to true to cause a heapdump to be created when a memory leak is detected .
Default true

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.leakSweeperDelay
Description Delay after an application or module stops to check for classloader leaks.
Default 10000 (milliseconds)

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.monitorSystemApps
Description Set to true to watch for memory leaks on system applications that are shipped by IBM , such as the administrative console.
Default false

We can fine tune the leak action policy behavior using the following thread and timer leaks JVM custom properties. The ThreadLocal, thread, timer and static leak custom properties are applicable only if com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks is set to true.

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesInterruptThreads
Description Set to true for WebSphere Application Server to attempt to interrupt threads that are started by the web application. Interrupting threads is performed using the Thread.interrupt() method. There is a possibility that the target thread might not respond to the interrupt.
Default true

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesHttpClientKeepAliveThread
Description If an HttpClient keep-alive timer thread is not started by this web application and is still running, WebSphere Application Server changes the context class loader from the current classloader to its parent to prevent a memory leak. The keep-alive timer thread stops on its own when the keep-alive threads all die. However, on a busy system that might not happen for some time.
Default true

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesStopTimerThreads
Description Set to true for WebSphere Application Server to stop any java.util.TimerThreads that are started by the web application.
Default false

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.jvmThreadGroupNames
Description List of ThreadGroup names to ignore when scanning for threads that are started by the web application that must be shut down. This list is delineated by underscores.
Default system__RMI Runtime

The following static class variable leaks properties apply only if com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks is set to true.

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesStatic
Description Set to true for WebSphere Application Server to attempt to make final fields null from loaded classes when a web application stops, as a work around for garbage collection bugs and application coding errors. Applications without memory leaks using recent JVMs should operate correctly when this option is set to false.
Default false

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.filterPrefixes
Description Member attributes starting with these filters are not set to null when clearReferencesStatic is true.
Default java javax com.ibm org sun com.sun. The list is delineated by spaces.

The following Threadlocal leaks properties apply only if com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks is set to true.

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.checkThreadLocalLeaks
Description Determines if ThreadLocal leaks should be checked when an application stops.
Default false

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.clearReferencesThreadLocal
Description Eliminate threadlocal leaks by renewing threads in the threadpool when an application stops. This property applies only to distributed platforms.
Default true

The following properties apply only if clearReferencesThreadLocal is set to true.

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.renewThreadPoolNames
Description Threadpool names that WebSphere Application Server must renew when a threadlocal leak is detected.
Default "WebContainer" list is delineated by spaces.

Information Value
Name com.ibm.ws.runtime.component.MemoryLeakConfig.threadPoolRenewalDelayFactor
Description Control the total amount of time to wait for thread pool to be renewed. The amount of time to wait for threadpool renewal = threadPoolRenewalDelayFactor * threadPool.getKeepAliveTime()
Default 10. For the web container threadpool, the renewal delay is (10 * 60000 ms) 6 minutes.

Information Value
Name com.ibm.ws.util.ThreadPool.DEFAULT_THREAD_RENEWAL_DELAY
Description Threads in the pool renew after an application stops. Use this custom property to avoid renewing all threads at the same time. This delay is observed between two threads being renewed.
Default 1000 (value is in ms). If the value is negative, threads are never renewed.

These JVM custom properties are persisted in the WAS configuration model in the server.xml file. The following code snippet displays the persisted leak policy configuration from the server_home/config/cells/nodes/servers/server.xml file of an unmanaged server:
<jvmEntries xmi:id="JavaVirtualMachine_1183122130078" 
            verboseModeClass="true" 
            verboseModeGarbageCollection="true" 
            verboseModeJNI="false"  
            runHProf="false" 
            hprofArguments="" 
            debugMode="false" 
            debugArgs="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777"  
            genericJvmArguments="-agentlib:getClasses 
            -Xquickstart  
            -Xalwaysclassgc" 
            executableJarFileName="" 
            disableJIT="false">
 <systemProperties xmi:id="Property_1317048628648" name="com.ibm.ws.runtime.component.MemoryLeakConfig.detectAppCLLeaks" value="true" />
 <systemProperties xmi:id="Property_1318975518491" name="com.ibm.ws.runtime.component.MemoryLeakConfig.clearAppCLLeaks" value="true" />
 <systemProperties xmi:id="Property_1318955284241" name="com.ibm.ws.runtime.component.MemoryLeakConfig.generateSystemDumps" value="false" />
 <systemProperties xmi:id="Property_1319119976147" name="com.ibm.ws.runtime.component.MemoryLeakConfig.generateHeapDumps" value="true" />
 <systemProperties xmi:id="Property_1317048628649" name="com.ibm.ws.runtime.component.MemoryLeakConfig.monitorSystemApps" value="false" />
</jvmEntries>
Click Apply.

Click OK.

Save the changes. Make sure that a file synchronization is performed before restarting the servers.

Restart the Application Server for the changes to take effecte
Example

The Memory Leak policy for WAS can be configured and persisted using JVM custom properties as described in this example. At runtime the memory leak detection, prevention and policy configuration can be changed using the MemoryLeakConfig mbean.
Administration of the Memory leak policy can be carried out using the MemoryLeakAdmin mbean. The leak policy affects how the application server responds to a classloader memory leak when an application or when the server is stopped.
We can adjust the Memory leak policy settings using the wsadmin scripting interface. These changes take effect immediately, but do not persist to the server configuration, and are lost when the server is restarted. The following script provides an example of how to configure and administer the memory leak policy using wsadmin jacl scripting :
# Scripting in JACL
 # Get the object name of the MemoryLeak Configiration object  to change the values on set leakConfig [$AdminControl completeObjectName "type=MemoryLeakConfig,*"]
WebSphere:cell=smitaNode03Cell,name=LeakConfig,type=MemoryLeakConfig,node=smitaNode03,process=server1
 # Get the object name of the MemoryLeak Administration object  to issue operations set leakAdmin [$AdminControl completeObjectName "type=MemoryLeakAdmin,*"]
WebSphere:cell=smitaNode03Cell,name=LeakAdmin,type=MemoryLeakAdmin,node=smitaNode03,process=server1
 # Look at all the attributes of the MemoryLeakConfig mbean
wsadmin>$Help all $leakConfig
Name: WebSphere:cell=smitaNode03Cell,name=LeakConfig,type=MemoryLeakConfig,node=smitaNode03,process=server1
        Description: Information on the management interface of the MBean
        Class name: com.ibm.ws.runtime.component.MemoryLeakConfig
 Attribute                       Type                            Access 
JvmThreadGroupNames             java.lang.String                RW
FilterPrefixes                  java.lang.String                RW
RenewThreadPoolNames            java.lang.String                RW
DetectAppCLLeaks                boolean                         RW
ClearAppCLLeaks                 boolean                         RW
MonitorSystemApps               boolean                         RW
NoDumps                         boolean                         RW
GenerateHeapDumps               boolean                         RW
GenerateSystemDumps             boolean                         RW
ClearReferencesStatic           boolean                         RW
ClearReferencesInterruptThreads   boolean                       RW
ClearReferencesStopTimerThreads   boolean                       RW
ClearReferencesHttpClientKeepAliveThread  boolean               RW
ClearReferencesThreadLocal      boolean                         RW
LeakSweeperDelay                int                             RW
ThreadPoolRenewalDelayFactor   int                             RW
PreventJreMemoryLeaks           boolean                         RW
LeakConfiguration               java.lang.String                RO
 Operation
Notifications
Constructors
# Print the current Memory Leak Policy configuration on the console  
wsadmin>$AdminControl getAttribute $leakConfig LeakConfiguration

MemoryLeakConfig [getClass()=class com.ibm.ws.runtime.component.MemoryLeakConfig, hashCode()=37266644
    preventJreMemoryLeaks true 
    detectAppCLLeaks true          
    monitorSystemApps false          
    leakSweeperDelay 10000
    clearAppCLLeaks true          
    clearReferencesStopTimerThreads false          
    clearReferencesHttpClientKeepAliveThread true          
    clearReferencesInterruptThreads true                   
    jvmThreadGroupNames [system, RMI Runtime]
    clearReferencesStatic true                  
    filterPrefixes [java., javax., com.ibm., org., sun., com.sun]
    clearReferencesThreadLocal true                   
    renewThreadPoolNames [WebContainer]
    threadPoolRenewalDelayFactor 1
    noDumps false          
    generateHeapDumps true          
    generateSystemDumps false  
# Change the configuration 
wsadmin>$AdminControl setAttribute $leakConfig ThreadPoolRenewalDelayFactor 10
wsadmin>$AdminControl setAttribute $leakConfig ClearReferencesStopTimerThreads true  
# See the updated configuration 
wsadmin>$AdminControl getAttribute $leakConfig LeakConfiguration
MemoryLeakConfig [getClass()=class com.ibm.ws.runtime.component.MemoryLeakConfig, 
    hashCode()=37266644
    preventJreMemoryLeaks true 
    detectAppCLLeaks true          
    monitorSystemApps false          
    leakSweeperDelay 10000
    clearAppCLLeaks true          
    clearReferencesStopTimerThreads true          
    clearReferencesHttpClientKeepAliveThread true          
    clearReferencesInterruptThreads true                   
    jvmThreadGroupNames [system, RMI Runtime]
    clearReferencesStatic true                  
    filterPrefixes [java., javax., com.ibm., org., sun., com.sun]
    clearReferencesThreadLocal true                   
    renewThreadPoolNames [WebContainer]
    threadPoolRenewalDelayFactor 10
    noDumps false          
    generateHeapDumps true          
    generateSystemDumps false  
    # Look at all the operations of the MemoryLeakAdmin mbean
wsadmin>$Help all $leakAdmin
    Name: WebSphere:cell=smitaNode03Cell,name=LeakAdmin,type=MemoryLeakAdmin,node=smitaNode03,process=server1
    Description: Information on the management interface of the MBean
    Class name: com.ibm.ws.runtime.component.MemoryLeakAdmin
    Operation
    java.lang.String findLeaks()
    java.lang.String fixLeaks()
    java.lang.String fixLeaks(java.lang.String)
# Find current classloader memory leaks
wsadmin>$AdminControl invoke $leakAdmin findLeaks
    CWMML0028I: The following web applications were stopped (reloaded, undeployed),  
    but their classes from previous runs are still loaded in memory,  
    thus causing a memory leak.[[78577.075.724.NWALogging#NWALoggingEJB.jar]].
# Fix ALL current classloader memory leaks
wsadmin>$AdminControl  invoke $leakAdmin fixLeaks
CWMML0036I: Please watch the SystemOut log for results of the fix leak operation.
 wsadmin>$AdminControl  invoke $leakAdmin fixLeaks {"78577.075.724.NWALogging#NWALoggingEJB.jar"}
CWMML0036I: Please watch the SystemOut log for results of the fix leak operation.
Subtopics

Memory leaks in JEE applications
Memory leaks come in various types such as thread and ThreadLocal leaks, ClassLoader leaks, system resource leaks, and connection leaks. Approaches to memory leak detection typically involve examination of Java virtual machine tool Interface (JVMTI) or performance monitoring infrastructure (PMI) counters to watch for slow growth in Java or native heap use.

Java virtual machine custom properties

Information	Value
Name	com.ibm.ws.runtime.component.MemoryLeakConfig.detectAppCLLeaks
Description	When the server is shutting down or an application stops, WAS determines the classloaders that have leaked and issues warnings and other additional information that aids in debugging the memory leak. See also the Improved classloader leak detection PMR.
Default	false

Information	Value
Name	com.ibm.ws.util.ThreadPool.DEFAULT_THREAD_RENEWAL_DELAY
Description	Threads in the pool renew after an application stops. Use this custom property to avoid renewing all threads at the same time. This delay is observed between two threads being renewed.
Default	1000 (value is in ms). If the value is negative, threads are never renewed.