Using the WebLogic 8.1 Thread Pool Model

Performance and Tuning

Using the WebLogic 8.1 Thread Pool Model

If you have been using execute queues to improve performance prior to this release, you may continue to use them after you upgrade your application domains to WebLogic Server 9.x.

Oracle recommends migrating from execute queues to using the self-tuning execute queue with work managers. See Using Work Managers to Optimize Scheduled Work in Configuring WebLogic Server Environments.

How to Enable the WebLogic 8.1 Thread Pool Model
Tuning the Default Execute Queue
Using Execute Queues to Control Thread Usage
Monitoring Execute Threads
Allocating Execute Threads to Act as Socket Readers
Tuning the Stuck Thread Detection Behavior

How to Enable the WebLogic 8.1 Thread Pool Model

Oracle provides a flag that enables you to disable the self-tuning execute pool and provide backward compatibility for upgraded applications to continue to use user-defined execute queues.
To use user-defined execute queues in a WebLogic Server 9.x domain, you need to include the use81-style-execute-queues sub-element of the server element in the config.xml file and reboot the server.

To run WebLogic Server in the 81-style-execute-queues mode, and be able to tune the thread count, an additional step is needed. First, set the use81-style-execute-queues element to true in the configuration file. Then, explicitly create the weblogic.kernel.Default execute queue from the Administration Console and reboot the server.
The following example code allows an instance of myserver to use execute queues. Listing B-1 Using the use81-style-execute-queues Element
.

.

.

<server>

   <name>myserver</name>

   <ssl>

      <name>myserver</name>

      <enabled>true</enabled>

      <listen-port>7002</listen-port>

   </ssl>

   <use81-style-execute-queues>true</use81-style-execute-queues>

   <listen-address/>

</server>
.

.

.
Configured work managers are converted to execute queues at runtime by the server instance.
Tuning the Default Execute Queue

The value of the ThreadCount attribute of an ExecuteQueue element in the config.xml file equals the number of simultaneous operations that can be performed by applications that use the execute queue. As work enters an instance of WebLogic Server, it is placed in an execute queue. This work is then assigned to a thread that does the work on it. Threads consume resources, so handle this attribute with care—you can degrade performance by increasing the value unnecessarily. WebLogic Server uses different default values for the thread count of the default execute queue depending on the startup mode of the server instance. See “ Specify a startup mode” in Administration Console Online Help.

Table B-1 Default Thread Count for Startup Modes
Server Mode . . . Default Thread Count . . .
Development 15 threads
Production
25 threads

Unless you configure additional execute queues, and assign applications to them, the server instance assigns requests to the default execute queue.

If native performance packs are not being used for your platform, you may need to tune the default number of execute queue threads and the percentage of threads that act as socket readers to achieve optimal performance. For more information, see Allocating Execute Threads to Act as Socket Readers.

Should You Modify the Default Thread Count?

Adding more threads to the default execute queue does not necessarily imply that you can process more work. Even if you add more threads, you are still limited by the power of your processor. You can degrade performance by increasing the value of the ThreadCount attribute unnecessarily. A high execute thread count causes more memory to be used and may increases context switching, which can degrade performance.
The value of the ThreadCount attribute depends very much on the type of work your application does. For example, if your client application is thin and does a lot of its work through remote invocation, that client application will spend more time connected — and thus will require a higher thread count — than a client application that does a lot of client-side processing.
If you do not need to use more than 15 threads (the development default) or 25 threads (the production default) for your work, do not change the value of this attribute. As a general rule, if your application makes database calls that take a long time to return, you will need more execute threads than an application that makes calls that are short and turn over very rapidly. For the latter case, using a smaller number of execute threads could improve performance.
To determine the ideal thread count for an execute queue, monitor the queue's throughput while all applications in the queue are operating at maximum load. Increase the number of threads in the queue and repeat the load test until you reach the optimal throughput for the queue. (At some point, increasing the number of threads will lead to enough context switching that the throughput for the queue begins to decrease.)

The WebLogic Server Administration Console displays the cumulative throughput for all of a server's execute queues. To access this throughput value, follow steps 1-6 in Using Execute Queues to Control Thread Usage.
Table B-2 shows default scenarios for adjusting available threads in relation to the number of CPUs available in the WebLogic Server domain. These scenarios also assume that WebLogic Server is running under maximum load, and that all thread requests are satisfied by using the default execute queue. If you configure additional execute queues and assign applications to specific queues, monitor results on a pool-by-pool basis.

Table B-2 Scenarios for Modifying the Default Thread Count
When... And you see... Do This:
Thread Count < number of CPUs CPUs are under utilized, but there is work that could be done. Increase the thread count.
Thread Count = number of CPUs CPUs are under utilized, but there is work that could be done. Increase the thread count.
Thread Count > number of CPUs (by a moderate number of threads) CPUs have high utilization, with a moderate amount of context switching. Tune the moderate number of threads and compare performance results.
Thread Count > number of CPUs (by a large number of threads) Too much context switching. Reduce the number of threads.

Using Execute Queues to Control Thread Usage

You can fine-tune an application's access to execute threads (and thereby optimize or throttle its performance) by using user-defined execute queues in WebLogic Server. However, keep in mind that unused threads represent significant wasted resources in a WebLogic Server system. You may find that available threads in configured execute queues go unused, while tasks in other queues sit idle waiting for threads to become available. In such a situation, the division of threads into multiple queues may yield poorer overall performance than having a single, default execute queue.
Default WebLogic Server installations are configured with a default execute queue which is used by all applications running on the server instance. You may want to configure additional queues to:

Optimize the performance of critical applications. For example, you can assign a single, mission-critical application to a particular execute queue, guaranteeing a fixed number of execute threads. During peak server loads, nonessential applications may compete for threads in the default execute queue, but the mission-critical application has access to the same number of threads at all times.
Throttle the performance of nonessential applications. For an application that can potentially consume large amounts of memory, assigning it to a dedicated execute queue effectively limits the amount of memory it can consume. Although the application can potentially use all threads available in its assigned execute queue, it cannot affect thread usage in any other queue.
Remedy deadlocked thread usage. With certain application designs, deadlocks can occur when all execute threads are currently utilized. For example, consider a servlet that reads messages from a designated JMS queue. If all execute threads in a server are used to process the servlet requests, then no threads are available to deliver messages from the JMS queue. A deadlock condition exists, and no work can progress. Assigning the servlet to a separate execute queue avoids potential deadlocks, because the servlet and JMS queue do not compete for thread resources.

Be sure to monitor each execute queue to ensure proper thread usage in the system as a whole. See Should You Modify the Default Thread Count? for general information about optimizing the number of threads.

Creating Execute Queues

An execute queue represents a named collection of execute threads that are available to one or more designated servlets, JSPs, EJBs, or RMI objects.
To configure a new execute queue using the Administration Console:

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the left pane of the console, expand Environment > Servers.
On the Summary of Servers page, select the server instance for which you will configure an execute queue.
Select the Configuration > Queues tab and click New.
Name of the execute queue and click OK.
On the User-Defined Execute Queues page, select the execute queue you just created.
On the execute queue Configuration tab, modify the following attributes or accept the system defaults:
Queue Length—Always leave the Queue Length at the default value of 65536 entries. The Queue Length specifies the maximum number of simultaneous requests that the server can hold in the queue. The default of 65536 requests represents a very large number of requests; outstanding requests in the queue should rarely, if ever reach this maximum value.
If the maximum Queue Length is reached, WebLogic Server automatically doubles the size of the queue to account for the additional work. Note, however, that exceeding 65536 requests in the queue indicates a problem with the threads in the queue, rather than the length of the queue itself; check for stuck threads or an insufficient thread count in the execute queue.
Queue Length Threshold Percent—The percentage (from 1– 99) of the Queue Length size that can be reached before the server indicates an overflow condition for the queue. All actual queue length sizes below the threshold percentage are considered normal; sizes above the threshold percentage indicate an overflow. When an overflow condition is reached, WebLogic Server logs an error message and increases the number of threads in the queue by the value of the Threads Increase attribute to help reduce the workload.
By default, the Queue Length Threshold Percent value is 90 percent. In most situations, you should leave the value at or near 90 percent, to account for any potential condition where additional threads may be needed to handle an unexpected spike in work requests. Keep in mind that Queue Length Threshold Percent must not be used as an automatic tuning parameter—the threshold should never trigger an increase in thread count under normal operating conditions.
Thread Count—The number of threads assigned to this queue. If you do not need to use more than 15 threads (the default) for your work, do not change the value of this attribute. (For more information, see Should You Modify the Default Thread Count?.)
Threads Increase—The number of threads WebLogic Server should add to this execute queue when it detects an overflow condition. If you specify zero threads (the default), the server changes its health state to “warning” in response to an overflow condition in the thread, but it does not allocate additional threads to reduce the workload.

If WebLogic Server increases the number of threads in response to an overflow condition, the additional threads remain in the execute queue until the server is rebooted. Monitor the error log to determine the cause of overflow conditions, and reconfigure the thread count as necessary to prevent similar conditions in the future. Do not use the combination of Threads Increase and Queue Length Threshold Percent as an automatic tuning tool; doing so generally results in the execute queue allocating more threads than necessary and suffering from poor performance due to context switching.
Threads Minimum—The minimum number of threads that WebLogic Server should maintain in this execute queue to prevent unnecessary overflow conditions. By default, the Threads Minimum is set to 5.
Threads Maximum—The maximum number of threads that this execute queue can have; this value prevents WebLogic Server from creating an overly high thread count in the queue in response to continual overflow conditions. By default, the Threads Maximum is set to 400.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes. Not all changes take effect immediately—some require a restart.
You must reboot the server to use the new thread detection behavior values.

Modifying the Thread Count

To modify the default execute queue thread count using the Administration Console:

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the left pane of the console, expand Environment > Servers.
On the Summary of Servers page, select the server instance for which you will configure thread detection behavior.
On the Configuration > Queues tab, select the execute queue for which you will modify the default thread count.

You can only modify the default execute queue for the server or a user-defined execute queue.
Locate the Thread Count value and increase or decrease it, as appropriate.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes. Not all changes take effect immediately—some require a restart.
You must reboot the server to use the new thread detection behavior values.

Tuning Execute Queues for Overflow Conditions

You can configure WebLogic Server to detect and optionally address potential overflow conditions in the default execute queue or any user-defined execute queue. WebLogic Server considers a queue to have a possible overflow condition when its current size reaches a user-defined percentage of its maximum size. When this threshold is reached, the server changes its health state to “warning” and can optionally allocate additional threads to perform the outstanding work in the queue, thereby reducing the queue length.
To automatically detect and address overflow conditions in a queue, you can configure the following items:

The threshold at which the server indicates an overflow condition. This value is set as a percentage of the configured size of the execute queue (the QueueLength value).
The number of threads to add to the execute queue when an overflow condition is detected. These additional threads work to reduce the size of the queue and reduce the size of the queue to its normal operating size.
The minimum and maximum number of threads available to the queue. In particular, setting the maximum number of threads prevents the server from assigning an overly high thread count in response to overload conditions.

To tune an execute queue using the WebLogic Server Administration Console:

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the left pane of the console, expand Environment > Servers.
On the Summary of Servers page, select the server instance for which you will configure overflow conditions behavior.
Select the Configuration > Queues tab, select the execute queue for which you will configure overflow conditions behavior.
Specify how the server instance should detect an overflow condition for the selected queue by modifying the following attributes:
Queue Length—Specifies the maximum number of simultaneous requests that the server can hold in the queue. The default of 65536 requests represents a very large number of requests; outstanding requests in the queue should rarely, if ever reach this maximum value. Always leave the Queue Length at the default value of 65536 entries.
Queue Length Threshold Percent—The percentage (from 1– 99) of the Queue Length size that can be reached before the server indicates an overflow condition for the queue. All actual queue length sizes below the threshold percentage are considered normal; sizes above the threshold percentage indicate an overflow. By default, the Queue Length Threshold Percent is set to 90 percent.
To specify how this server should address an overflow condition for the selected queue, modify the following attribute:
Threads Increase—The number of threads WebLogic Server should add to this execute queue when it detects an overflow condition. If you specify zero threads (the default), the server changes its health state to “warning” in response to an overflow condition in the execute queue, but it does not allocate additional threads to reduce the workload.
To fine-tune the variable thread count of this execute queue, modify the following attributes:
Threads Minimum—The minimum number of threads that WebLogic Server should maintain in this execute queue to prevent unnecessary overflow conditions. By default, the Threads Minimum is set to 5.
Threads Maximum—The maximum number of threads that this execute queue can have; this value prevents WebLogic Server from creating an overly high thread count in the queue in response to continual overflow conditions. By default, the Threads Maximum is set to 400.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes. Not all changes take effect immediately—some require a restart.
You must reboot the server to use the new thread detection behavior values.

Assigning Servlets and JSPs to Execute Queues

You can assign a servlet or JSP to a configured execute queue by identifying the execute queue name in the initialization parameters. Initialization parameters appear within the init-param element of the servlet's or JSP's deployment descriptor file, web.xml. To assign an execute queue, enter the queue name as the value of the wl-dispatch-policy parameter, as in the example:
<servlet>

   <servlet-name>MainServlet</servlet-name>

   <jsp-file>/myapplication/critical.jsp</jsp-file>

   <init-param>

      <param-name>wl-dispatch-policy</param-name>

      <param-value>CriticalAppQueue</param-value>

   </init-param>

</servlet>
See “ Creating and Configuring Servlets” in Developing Web Applications, Servlets, and JSPs for WebLogic Server for more information about specifying initialization parameters in web.xml.

Assigning EJBs and RMI Objects to Execute Queues

To assign an EJB object to a configured execute queue, use the new dispatch-policy element in weblogic-ejb-jar.xml. For more information, see the weblogic-ejb-jar.xml Deployment Descriptor.
While you can also set the dispatch policy through the appc compiler -dispatchPolicy flag, Oracle strongly recommends you use the deployment descriptor element instead. This way, if the EJB is recompiled, during deployment for example, the setting will not be lost.
To assign an RMI object to a configured execute queue, use the -dispatchPolicy option to the rmic compiler. For example:
java weblogic.rmic -dispatchPolicy CriticalAppQueue ...

Monitoring Execute Threads

To use the Administration Console to monitor the status of execute threads:

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the left pane of the console, expand Environment > Servers.
On the Summary of Servers page, select the server instance for which you will configure thread detection behavior.
Select the Monitoring > Threads tab.
A table of the execute queues available on this server instance is displayed.
Select an execute queue for which you would like to view thread information.
A table of execute threads for the selected execute queue is displayed.

Allocating Execute Threads to Act as Socket Readers

For best performance, Oracle recommends that you use the native socket reader implementation, rather than the pure-Java implementation, on machines that host WebLogic Server instances (see Thread Management). However, if use the pure-Java socket reader implementation for host machines, you can still improve the performance of socket communication by configuring the proper number of execute threads to act as socket reader threads for each server instance.
The ThreadPoolPercentSocketReaders attribute sets the maximum percentage of execute threads that are set to read messages from a socket. The optimal value for this attribute is application-specific. The default value is 33, and the valid range is 1– 99.
Allocating execute threads to act as socket reader threads increases the speed and the ability of the server to accept client requests. It is essential to balance the number of execute threads that are devoted to reading messages from a socket and those threads that perform the actual execution of tasks in the server.

Setting the Number of Socket Reader Threads For a Server Instance

To use the Administration Console to set the maximum percentage of execute threads that read messages from a socket:

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the left pane of the console, expand Environment > Servers.
On the Summary of Servers page, select the server instance for which you will configure thread detection behavior.
Select the Configuration > Tuning tab.
Specify the percentage of Java reader threads in the Socket Readers field. The number of Java socket readers is computed as a percentage of the number of total execute threads (as shown in the Thread Count field for the Execute Queue).
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes.

Setting the Number of Socket Reader Threads on Client Machines

On client machines, you can configure the number of available socket reader threads in the JVM that runs the client. Specify the socket readers by defining the following parameters in the java command line for the client:
-Dweblogic.ThreadPoolSize=value
-Dweblogic.ThreadPoolPercentSocketReaders=value

Tuning the Stuck Thread Detection Behavior

WebLogic Server automatically detects when a thread in an execute queue becomes “stuck.” Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread.
WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune a server's thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency which the server checks for stuck threads. Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the “warning” and “critical” health states when all threads in a particular execute queue become stuck. For more information, see “ Understanding WebLogic Logging Services” in Configuring Log Files and Filtering Log Messages.
To configure stuck thread detection behavior:

If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
In the left pane of the console, expand Environment > Servers.
On the Summary of Servers page, select the server instance for which you will configure thread detection behavior.
On the Configuration > Tuning tab, update as necessary:
Stuck Thread Max Time—Amount of time, in seconds, that a thread must be continually working before a server instance diagnoses a thread as being stuck.
Stuck Thread Timer Interval—Amount of time, in seconds, after which a server instance periodically scans threads to see if they have been continually working for the configured Stuck Thread Max Time.
Click Save.
To activate these changes, in the Change Center of the Administration Console, click Activate Changes. Not all changes take effect immediately—some require a restart.
You must reboot the server to use the new thread detection behavior values.

When...	And you see...	Do This:
Thread Count < number of CPUs	CPUs are under utilized, but there is work that could be done.	Increase the thread count.
Thread Count = number of CPUs	CPUs are under utilized, but there is work that could be done.	Increase the thread count.
Thread Count > number of CPUs (by a moderate number of threads)	CPUs have high utilization, with a moderate amount of context switching.	Tune the moderate number of threads and compare performance results.
Thread Count > number of CPUs (by a large number of threads)	Too much context switching.	Reduce the number of threads.