WAS v8.5 > TroubleshootDefault behavior for OutOfMemory exceptions
WebSphere Application Server v8 Fix Pack 2 and later ships with IBM Java 6 R26 on supported operating systems. Beginning with this version, the default behavior for OutOfMemory (OOM) exceptions has changed.
By default in IBM Java 6 R26 and later, the first OOM for the lifetime of a Java process produces a PHD-formatted heap dump, a Java dump file, a snap dump file, and an operating system dump-core file on Linux, AIX , and IBM i (not to be confused with a javacore), user-mode minidump with full memory on Windows operating systems. The second, third, and fourth OOM exceptions produce only a PHD-formatted heap dump and a Java dump file. Therefore, the change in default behavior is an additional system dump on the first OOM exception. A system dump is a superset of a PHD heap dump. A system dump also includes memory contents (strings, primitives, variable names, and so on), thread and frame local information, some native memory information, and more.
This added information can solve a larger class of problems, provide more general insight into a running JVM, and ultimately reduce the time that it takes to solve a problem. In IBM Java 5 R12 or earlier and IBM Java 6 R9 or earlier, a system dump had to be post-processed using the jextract tool. With recent versions of IBM Java, however, including the one that ships with this new OOM behavior, a system dump can be directly loaded by a DTFJ-capable tool such as the Memory Analyzer Tool without any post-processing for OOMs (just like a PHD heap dump). For crashes, however, jextract should still be used.
The default configuration can be observed with the -Xdump:what generic JVM argument. For example, running...
$WAS/java/bin/java -version -Xdump:what
...produces the following output (some output removed). Note the range option in particular.
# java -version -Xdump:what Registered dump agents ... -Xdump:system: events=systhrow, filter=java/lang/OutOfMemoryError, label=... core.&Y&m&d.&H% M% S.% pid.% seq.dmp, range=1..1, priority=999, request=exclusive+compact+prepwalk ... -Xdump:heap: events=systhrow, filter=java/lang/OutOfMemoryError, label=... heapdump.% Y% m% d.% H% M% S.% pid.% seq.phd, range=1..4, priority=500, request=exclusive+compact+prepwalk, opts=PHD ... -Xdump:java: events=systhrow, filter=java/lang/OutOfMemoryError, label=... javacore.% Y% m% d.% H% M% S.% pid.% seq.txt, range=1..4, priority=400, request=exclusive+preempt ... -Xdump:snap: events=systhrow, filter=java/lang/OutOfMemoryError, label=... Snap.% Y% m% d.% H% M% S.% pid.% seq.trc, range=1..4, priority=300, request=serialA blank space was added to this example after each occurrence of the % character to avoid inappropriate conversions of the text in this information center.
This configuration can be changed using the -Xdump generic JVM argument.
- To revert to the old behavior on OOM, use:
-Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache
- To remove PHD heap dumps on OOM, use:
-Xdump:heap:none
- To remove PHD heap dumps on OOM and take system dumps on not just the first OOM but on the first four:
-Xdump:heap:none -Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..4,request=exclusive+compact+prepwalk
It is critical that users ensure that operating systems are correctly configured to produce untruncated system dumps:
System dumps are written to the location specified by the file parameter in -Xdump (see the previous label attribute in -Xdump:what). You should change this to a dedicated partition with sufficient space. For example:
-Xdump:system:file=/var/dumps/core.% Y% m% d.% H% M% S.% pid.% seq.dmp
A blank space was added to this example after each occurrence of the % character to avoid inappropriate conversions of the text in this information center.
You might also consider using the -Xdiagnosticscollector generic JVM argument, which will print a warning to native_stderr.log if the JVM detects that ulimits are set incorrectly on startup.