Web module or appserver stops processing requests


 

+

Search Tips   |   Advanced Search

 

If an appserver process spontaneously closes, or Web modules running on the appserver stop responding to new requests:

  1. Isolate the problem by installing the Web modules on different servers, if possible.

  2. Check WAS directory structure for a file with a name like...

    javacore[number].txt

    This file is a Java thread dump file that the JVM creates if an appserver process spontaneously closes.

  3. Use the TPV to determine if any of the appserver resources, such as the Java heap, or database connections, have reached their maximum capacity.

    If there is a resource problem, review the application code for a possible cause:

    • If database connections are being assigned to a request but are not being released when the requests finish processing, ensure that the application code performs a close() on any opened Connection object within a finally{} block.

    • If there is a steady increase in servlet engine threads in use, review application synchronized code blocks for possible deadlock conditions.

    • If there is a steady increase in a JVM heap size, review application code for memory leak opportunities, such as static (class-level) collections, that can cause objects to never get garbage-collected.

  4. Enable verbose garbage collection on the appserver to help you determine if we have a memory leak problems.

    This feature adds detailed statements about the amount of available and in-use memory to the JVM error log file. To enable up verbose garbage collection:

    1. In the admin console, click...

    2. Stop and restart the appserver.

    3. Periodically, browse the log file for garbage collection statements. Look for statements beginning with "allocation failure". This string indicates that a need for memory allocation has triggered a JVM garbage collection, to release unused memory. Allocation failures are normal and do not necessarily indicate a problem. However, the statements that follow the allocation failure statement show how many bytes are needed and how many are allocated. If these bytes needed statements indicate that the JVM keeps allocating more memory for its own use, or that the JVM is unable to allocate as much memory as it needs, there might be a memory leak.

    We can also use the TPV to detect memory leak problems.

  5. Determine if the appserver is running out of memory.

    If we determine that the appserver is running out of memory, one of the following situations might be occurring:

    • There is a memory leak in application code that address.

      To pinpoint the cause of a memory leak, enable the RunHProf property on the Java Virtual Machine page of the admin console.

      • Set the HProf Arguments field to a value similar to...

        depth=20,file=heapdmp.txt

        This value shows exception stacks to a maximum of 20 levels, and saves the heapdump output to...

        APP_ROOT/bin/heapdmp.txt

      • Save the settings.

      • Stop and restart the appserver.

      • If possible, reenact the scenario or access the resource that caused the appserver's process to spontaneously close, or its Web modules to stop responding to new requests.

        Then stop the appserver. If we cannot reenact the scenario or access the resource, wait until the problem reoccurs, and then stop the appserver.

      • Examine the file into which the heap dump was saved. For example, examine the APP_ROOT/bin/heapdmp.txt file:

        • Search for the string, "SITES BEGIN". This finds the location of a list of Java objects in memory, which shows the amount of memory allocated to the objects.

        • The list of Java objects occurs each time there was a memory allocation in the JVM.

          There is a record of what type of object the memory instantiated and an identifier of a trace stack, listed elsewhere in the dump, that shows the Java method that made the allocation.

        • The list of Java object is in descending order by number of bytes allocated.

          Depending on the nature of the leak, the problem class should show up near the top of the list, but this is not always the case. Look throughout the list for large amounts of memory or frequent instances of the same class being instantiated. In the latter case, use the ID in the trace stack column to identify allocations occurring repeatedly in the same class and method.

        • Examine the source code indicated in the related trace stacks for the possibility of memory leaks.

    • The JVM is using the maximum heap size that it is allowed to use.

      In this situation, you should increase the maximum heap size setting for appserver if we have enough storage available to do so.

    • The server runtime is experiencing a problem.

      If we determine that there is a problem with the server runtime, make sure that we have applied all of the service updates for WAS ND. If, after you apply all of the service updates, the problem still exists, contact IBM Support.

  6. Browse the thread dump for clues:

    The JVM creates a thread dump whenever an appserver process spontaneously closes. We can also force an application to create a thread dump. After a dump is created, we can check the dump for clues as to why new requests are not being processed.

    To force a thread dump:

    1. Use the wsadmin command prompt, get a handle to the problem appserver:

      wsadmin>set jvm [$AdminControl completeObjectName type=JVM,process=server_name,*]
      wsadmin>$AdminControl invoke $jvm dumpThreads

    2. Look for an output file, in the installation root directory for WAS ND, with a name like...

      javacore.date.time.id.txt.

    After the application creates the dump, we can check for the following clues:

    • "Error" or "exception information" strings at the beginning of the file. These strings indicate the thread that caused the appserver process to spontaneously close. These strings are not be present if we forced the dump.

    • Look at the snapshot of each thread in the process. The thread dump contains a snapshot of each thread in the process, starting in the section labeled "Full thread dump."

      • Look for threads with a description that contains "state:R". Such threads are active and running when the dump is forced, or the process exited.

      • Look for multiple threads in the same Java application code source location. Multiple threads from the same location might indicate a deadlock condition (multiple threads waiting on a monitor) or an infinite loop, and help identify the application code with the problem.

IBM Support has documents and tools that can save you time gathering information needed to resolve problems as described in Troubleshooting help from IBM.


 

Related tasks

Monitor performance with Tivoli Performance Viewer
Start the wsadmin scripting client
Manage appservers