Performance and Tuning

Tuning WebLogic Server

The following sections describe how to tune WebLogic Server to match your application needs.

• Setting Java Parameters for Starting WebLogic Server

• Development vs. Production Mode Default Tuning Values

• Thread Management

• Tuning Network I/O

• Setting Your Compiler Options

• Using WebLogic Server Clusters to Improve Performance

• Monitoring a WebLogic Server Domain

 

---

Setting Java Parameters for Starting WebLogic Server

Java parameters must be specified whenever you start WebLogic Server. For simple invocations, this can be done from the command line with the weblogic.Server command. However, because the arguments needed to start WebLogic Server from the command line can be lengthy and prone to error, Oracle recommends that you incorporate the command into a script. To simply this process, you can modify the default values in the sample scripts that are provided with the WebLogic distribution to start WebLogic Server, as described in “ Specifying Java Options for a WebLogic Server Instance”.

If you used the Configuration Wizard to create your domain, the WebLogic startup scripts are located in the domain-name directory where you specified your domain. By default, this directory is BEA_HOME\user_projects\domain\domain-name, where BEA_HOME is the directory that contains the product installation, and domain-name is the name of the domain directory defined by the selected configuration template. For more information about creating domains using the Configuration Wizard, see “ Creating Domains Using the Configuration Wizard”.

You need to modify some default Java values in these scripts to fit your environment and applications. The important performance tuning parameters in these files are the JAVA_HOME parameter and the Java heap size parameters:

• Change the value of the variable JAVA_HOME to the location of your JDK. For example:

  set JAVA_HOME=C:\bea\jdk150_03

• For higher performance throughput, set the minimum java heap size equal to the maximum heap size. For example:

  "%JAVA_HOME%\bin\java" -server – Xms512m – Xmx512m -classpath %CLASSPATH% -

  See Specifying Heap Size Values for details about setting heap size options.

 

---

Development vs. Production Mode Default Tuning Values

You can indicate whether a domain is to be used in a development environment or a production environment. WebLogic Server uses different default values for various services depending on the type of environment you specify. Specify the startup mode for your domain as shown in the following table.

Table 6-1 Startup Modes

Choose this mode	when . . .
Development	You are creating your applications. In this mode, the configuration of security is relatively relaxed, allowing you to auto-deploy applications.
Production	Your application is running in its final form. In this mode, security is fully configured.

Table 6-2 lists the performance-related configuration parameters that differ when switching from development to production startup mode.

Table 6-2 Differences Between Development and Production Modes

Tuning Parameter	In development mode . . .	In production mode . . .
SSL	You can use the demonstration digital certificates and the demonstration keystores provided by the WebLogic Server security services. With these certificates, you can design your application to work within environments secured by SSL. For more information about managing security, see “ Configuring SSL” in Securing WebLogic Server.	You should not use the demonstration digital certificates and the demonstration keystores. If you do so, a warning message is displayed.
Deploying Applications	WebLogic Server instances can automatically deploy and update applications that reside in the domain_name/autodeploy directory (where domain_name is the name of a domain). It is recommended that this method be used only in a single-server development environment. For more information, see “ Auto-Deploying Applications in Development Domains” in Deploying Applications to WebLogic Server.	The auto-deployment feature is disabled, so use the WebLogic Server Administration Console, the weblogic.Deployer tool, or the WebLogic Scripting Tool (WLST). For more information, see Deploying Applications to WebLogic Server.

For information on switching the startup mode from development to production, see Change to Production Mode in the Administration Console Online Help

 

---

Thread Management

WebLogic Server provides the following mechanisms to manage threads to perform work.

• Tuning a Work Manager

• Tuning Execute Queues

• Understanding the Differences Between Work Managers and Execute Queues

• Tuning the Stuck Thread Detection Behavior

 

Tuning a Work Manager

In this release, WebLogic Server allows you to configure how your application prioritizes the execution of its work. Based on rules you define and by monitoring actual runtime performance, WebLogic Server can optimize the performance of your application and maintain service level agreements (SLA).

You tune the thread utilization of a server instance by defining rules and constraints for your application by defining a Work Manger and applying it either globally to WebLogic Server domain or to a specific application component. The primary tuning considerations are:

• How Many Work Managers are Needed?

• What are the SLA Requirements for Each Work Manager?

See Using Work Managers to Optimize Scheduled Work in Configuring WebLogic Server Environments.

How Many Work Managers are Needed?

Each distinct SLA requirement needs a unique work manager.

What are the SLA Requirements for Each Work Manager?

Service level agreement (SLA) requirements are defined by instances of request classes. A request class expresses a scheduling guideline that a server instance uses to allocate threads. See “ Understanding Work Managers” in Configuring WebLogic Server Environments.

 

Tuning Execute Queues

Execute Queues are deprecated in this release of WebLogic Server. Oracle recommends migrating applications to use work managers.

In previous versions of WebLogic Server, processing was performed in multiple execute queues. Different classes of work were executed in different queues, based on priority and ordering requirements, and to avoid deadlocks. See Using the WebLogic 8.1 Thread Pool Model.

 

Understanding the Differences Between Work Managers and Execute Queues

The easiest way to conceptually visualize the difference between the execute queues of previous releases with work managers is to correlate execute queues (or rather, execute-queue managers) with work managers and decouple the one-to-one relationship between execute queues and thread-pools.

For releases prior to WebLogic Server 9.0, incoming requests are put into a default execute queue or a user-defined execute queue. Each execute queue has an associated execute queue manager that controls an exclusive, dedicated thread-pool with a fixed number of threads in it. Requests are added to the queue on a first-come-first-served basis. The execute-queue manager then picks the first request from the queue and an available thread from the associated thread-pool and dispatches the request to be executed by that thread.

For releases of WebLogic Server 9.0 and higher, there is a single priority-based execute queue in the server. Incoming requests are assigned an internal priority based on the configuration of work managers you create to manage the work performed by your applications. The server increases or decreases threads available for the execute queue depending on the demand from the various work-managers. The position of a request in the execute queue is determined by its internal priority:

• The higher the priority, closer it is placed to the head of the execute queue.

• The closer to the head of the queue, more quickly the request will be dispatched a thread to use.

Work managers provide you the ability to better control thread utilization (server performance) than execute-queues, primarily due to the many ways that you can specify scheduling guidelines for the priority-based thread pool. These scheduling guidelines can be set either as numeric values or as the capacity of a server-managed resource, like a JDBC connection pool.

Migrating from Previous Releases

If you upgrade application domains from prior releases that contain execute queues, the resulting 9.x domain will contain execute queues.

• Migrating application domains from a previous release to WebLogic Server 9.x does not automatically convert an execute queues to work manager.

• If execute queues are present in the upgraded application configuration, the server instance assigns work requests appropriately to the execute queue specified in the dispatch-policy.

• Requests without a dispatch-policy use the self-tuning thread pool.

For more information on migrating a domain, see Upgrading WebLogic Application Environments.

 

Tuning the Stuck Thread Detection Behavior

WebLogic Server automatically detects when a thread in an execute queue becomes “stuck.” Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread.

WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune a server's thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency which the server checks for stuck threads. Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the “warning” and “critical” health states when all threads in a particular execute queue become stuck. For more information, see “ Configuring WebLogic Server to Avoid Overload Conditions” in Configuring WebLogic Server Environments. To configure stuck thread detection behavior, see “ Tuning execute thread detection behavior” in Administration Console Online Help.

 

---

Tuning Network I/O

The following sections provide information on network communication between clients and servers (including T3 and IIOP protocols, and their secure versions):

• Tuning Muxers

• Which Platforms Have Performance Packs?

• Enabling Performance Packs

• Changing the Number of Available Socket Readers

• Network Channels

• Tuning Message Size

• Tune the Chunk Parameters

• Tuning Connection Backlog Buffering

 

Tuning Muxers

WebLogic Server uses software modules called muxers to read incoming requests on the server and incoming responses on the client. These muxers are of two primary types: the Java muxer or native muxer.

A Java muxer has the following characteristics:

• Uses pure Java to read data from sockets.

• It is also the only muxer available for RMI clients.

• Blocks on reads until there is data to be read from a socket. This behavior does not scale well when there are a large number of sockets and/or when data arrives infrequently at sockets. This is typically not an issue for clients, but it can create a huge bottleneck for a server.

Native muxers use platform-specific native binaries to read data from sockets. The majority of all platforms provide some mechanism to poll a socket for data. For example, Unix systems use the poll system call and the Windows architecture uses completion ports. Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

If the Enable Native IO parameter is not selected, the server instance exclusively uses the Java muxer. This maybe acceptable if there are a small number of clients and the rate at which requests arrive at the server is fairly high. Under these conditions, the Java muxer performs as well as a native muxer and eliminate Java Native Interface (JNI) overhead. Unlike native muxers, the number of threads used to read requests is not fixed and is tunable for Java muxers by configuring the Percent Socket Readers parameter setting in the Administration Console. See Changing the Number of Available Socket Readers. Ideally, you should configure this parameter so the number of threads roughly equals the number of remote concurrently connected clients up to 50% of the total thread pool size. Each thread waits for a fixed amount of time for data to become available at a socket. If no data arrives, the thread moves to the next socket.

With native muxers, you may be able to improve throughput for some cpu-bound applications (for example, SpecJAppServer) by using the following:

   -Dweblogic.socket.SocketMuxer.DELAY_POLL_WAKEUP=xx

where xx is the amount of time, in microseconds, to delay before checking if data is available. The default value is 0, which corresponds to no delay.

 

Which Platforms Have Performance Packs?

Benchmarks show major performance improvements when you use native performance packs on machines that host WebLogic Server instances. Performance packs use a platform-optimized, native socket multiplexor to improve server performance. For example, the native socket reader multiplexor threads have their own execute queue and do not borrow threads from the default execute queue, which frees up default execute threads to do application work

To see which platforms currently have performance packs available:

1. Go to the Certifications Pages.

2. Select your platform from the list of certified platforms.

 

Enabling Performance Packs

The use of native performance packs are enabled by default in the configuration shipped with your distribution. You can use the Administration Console to verify that performance packs are enabled. See “ Enable native IO” in Administration Console Online Help.

 

Changing the Number of Available Socket Readers

If use the pure-Java socket reader implementation for host machines, you can improve the performance of socket communication by configuring the proper number of socket reader threads for each server instance and client machine. See “ Tuning the number of available socket readers” in Administration Console Online Help.

 

Network Channels

Network channels, also called network access points, allow you to specify different quality of service (QOS) parameters for network communication. Each network channel is associated with its own exclusive socket using a unique IP address and port. By default, requests from a multi-threaded client are multiplexed over the same remote connection and the server instance reads requests from the socket one at a time. If the request size is large, this becomes a bottleneck.

Although the primary role of a network channel is to control the network traffic for a server instance, you can leverage the ability to create multiple custom channels to allow a multi-threaded client to communicate with server instance over multiple connections, reducing the potential for a bottleneck. To configure custom multi-channel communication, use the following steps:

1. Configure multiple network channels using different IP and port settings. See “ Configure custom network channels” in Administration Console Online Help.

2. In your client-side code, use a JNDI URL pattern similar to the pattern used in clustered environments. The following is an example for a client using two network channels:

   t3://<ip1>:<port1>,<ip2>:<port2>
   

See “ Understanding Network Channels” in Configuring WebLogic Server Environments.

 

Tuning Message Size

WebLogic Server allows you to specify a maximum incoming request size to reduce the potential for Denial of Service (DoS) attacks by preventing a server from being bombarded by a series of large requests. You can set a global value or set specific values for different protocols and network channels. Although it does not directly impact performance, JMS applications that aggregate messages before sending to a destination may be refused if the aggregated size is greater than specified value. See “ Servers: Protocols: General” in Administration Console Online Help and Tuning Applications Using Unit-of-Order.

 

Tune the Chunk Parameters

A chunk is a unit of memory that the WebLogic Server network layer, both on the client and server side, uses to read data from and write data to sockets. To reduce memory allocation costs, a server instance maintains a pool of these chunks. For applications that handle large amounts of data per request, increasing the value on both the client and server sides can boost performance. The default chunk size is about 4K. Use the following properties to tune the chunk size and the chunk pool size:

• weblogic.Chunksize—Sets the size of a chunk (in bytes). The primary situation in which this may need to be increased is if request sizes are large. It should be set to values that are multiples of the network's maximum transfer unit (MTU), after subtracting from the value any Ethernet or TCP header sizes. Set this parameter to the same value on the client and server.

• weblogic.utils.io.chunkpoolsize—Sets the maximum size of the chunk pool. The default value is 2048. The value may need to be increased if the server starts to allocate and discard chunks in steady state. To determine if the value needs to be increased, monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk.

• weblogic.PartitionSize—Sets the number of pool partitions used (default is 4). The chunk pool can be a source of significant lock contention as each request to access to the pool must be synchronized. Partitioning the thread pool spreads the potential for contention over more than one partition.

 

Tuning Connection Backlog Buffering

You can tune the number of connection requests that a WebLogic Server instance will accept before refusing additional requests. The Accept Backlog parameter specifies how many Transmission Control Protocol (TCP) connections can be buffered in a wait queue. This fixed-size queue is populated with requests for connections that the TCP stack has received, but the application has not accepted yet. For more information on TCP tuning, see Basic OS Tuning Concepts.

You can tune the number of connection requests that a WebLogic Server instance will accept before refusing additional requests, see “ Tune connection backlog buffering” in Administration Console Online Help.

 

---

Setting Your Compiler Options

You may improve performance by tuning your server's compiler options.

 

Compiling EJB Classes

Use the weblogic.appc utility to compile EJB container classes. If you compile Jar files for deployment into the EJB container, use weblogic.appc to generate the container classes. By default, ejbc uses the javac compiler. You may be able to improve performance by specifying a different compiler (such as IBM Jikes) using the -compiler flag or using the Administration console. For more information, see:

• “Implementing Enterprise Java Beans” in Programming WebLogic EJB

• Configure compiler options in Administration Console Online Help.

 

Setting JSP Compiler Options

WebLogic Server uses Javelin to compile JSPs. In the weblogic.xml file, the jsp-descriptor element defines parameter names and values for servlet JSPs. Use the precompile parameter to configure WebLogic Server to precompile your JSPs when WebLogic Server starts up. See the jsp-descriptor element.

If you receive the following error message received when compiling JSP files on a UNIX machine:

failed: java.io.IOException: Not enough space

Try any or all of the following solutions:

• Add more RAM if you have only 256 MB.

• Raise the file descriptor limit, for example:

  set rlim_fd_max = 4096
  

  set rlim_fd_cur = 1024
  

 

---

Using WebLogic Server Clusters to Improve Performance

A WebLogic Server cluster is a group of WebLogic Servers instances that together provide fail-over and replicated services to support scalable high-availability operations for clients within a domain. A cluster appears to its clients as a single server but is in fact a group of servers acting as one to provide increased scalability and reliability.

A domain can include multiple WebLogic Server clusters and non-clustered WebLogic Server instances. Clustered WebLogic Server instances within a domain behave similarly to non-clustered instances, except that they provide failover and load balancing. The Administration Server for the domain manages all the configuration parameters for the clustered and non-clustered instances.

For more information about clusters, see “ Understanding WebLogic Server Clustering”.

 

Scalability and High Availability

Scalability is the ability of a system to grow in one or more dimensions as more resources are added to the system. Typically, these dimensions include (among other things), the number of concurrent users that can be supported and the number of transactions that can be processed in a given unit of time.

Given a well-designed application, it is entirely possible to increase performance by simply adding more resources. To increase the load handling capabilities of WebLogic Server, add another WebLogic Server instance to your cluster—without changing your application. Clusters provide two key benefits that are not provided by a single server: scalability and availability.

WebLogic Server clusters bring scalability and high-availability to Java EE applications in a way that is transparent to application developers. Scalability expands the capacity of the middle tier beyond that of a single WebLogic Server or a single computer. The only limitation on cluster membership is that all WebLogic Servers must be able to communicate by IP multicast. New WebLogic Servers can be added to a cluster dynamically to increase capacity.

A WebLogic Server cluster guarantees high-availability by using the redundancy of multiple servers to insulate clients from failures. The same service can be provided on multiple servers in a cluster. If one server fails, another can take over. The ability to have a functioning server take over from a failed server increases the availability of the application to clients.

Caution:

Provided that you have resolved all application and environment bottleneck issues, adding additional servers to a cluster should provide linear scalability. When doing benchmark or initial configuration test runs, isolate issues in a single server environment before moving to a clustered environment.

Clustering in the Messaging Service is provided through distributed destinations; connection concentrators, and connection load-balancing (determined by connection factory targeting); and clustered Store-and-Forward (SAF). Client load-balancing with respect to distributed destinations is tunable on connection factories. Distributed destination Message Driven Beans (MDBs) that are targeted to the same cluster that hosts the distributed destination automatically deploy only on cluster servers that host the distributed destination members and only process messages from their local destination. Distributed queue MDBs that are targeted to a different server or cluster than the host of the distributed destination automatically create consumers for every distributed destination member. For example, each running MDB has a consumer for each distributed destination queue member.

 

How to Ensure Scalability for WebLogic Clusters

In general, any operation that requires communication between the servers in a cluster is a potential scalability hindrance. The following sections provide information on issues that impact the ability to linearly scale clustered WebLogic servers:

• Database Bottlenecks

• Session Replication

• Invalidation of Entity EJBs

• Invalidation of HTTP sessions

• JNDI Binding, Unbinding and Rebinding

Database Bottlenecks

In many cases where a cluster of WebLogic servers fails to scale, the database is the bottleneck. In such situations, the only solutions are to tune the database or reduce load on the database by exploring other options. See DataBase Tuning and Tuning JDBC Applications.

Session Replication

User session data can be stored in two standard ways in a Java EE application: stateful session EJBs or HTTP sessions. By themselves, they are rarely a impact cluster scalability. However, when coupled with a session replication mechanism required to provide high-availability, bottlenecks are introduced. If a Java EE application has Web and EJB components, you should store user session data in HTTP sessions:

• HTTP session management provides more options for handling fail-over, such as replication, a shared DB or file.

• Superior scalability.

• Replication of the HTTP session state occurs outside of any transactions. Stateful session bean replication occurs in a transaction which is more resource intensive.

• The HTTP session replication mechanism is more sophisticated and provides optimizations a wider variety of situations than stateful session bean replication.

See Session Management.

Invalidation of Entity EJBs

This applies to entity EJBs that use a concurrency strategy of Optimistic or ReadOnly with a read-write pattern.

Optimistic—When an Optimistic concurrency bean is updated, the EJB container sends a multicast message to other cluster members to invalidate their local copies of the bean. This is done to avoid optimistic concurrency exceptions being thrown by the other servers and hence the need to retry transactions. If updates to the EJBs are frequent, the work done by the servers to invalidate each other's local caches become a serious bottleneck. A flag called cluster-invalidation-disabled (default false) is used to turn off such invalidations. This is set in the rdbms descriptor file.

ReadOnly with a read-write pattern—In this pattern, persistent data that would otherwise be represented by a single EJB are actually represented by two EJBs: one read-only and the other updateable. When the state of the updateable bean changes, the container automatically invalidates corresponding read-only EJB instance. If updates to the EJBs are frequent, the work done by the servers to invalidate the read-only EJBs becomes a serious bottleneck.

Invalidation of HTTP sessions

Similar to Invalidation of Entity EJBs, HTTP sessions can also be invalidated. This is not as expensive as entity EJB invalidation, since only the session data stored in the secondary server needs to be invalidated. Oracle advises users to not invalidate sessions unless absolutely required.

JNDI Binding, Unbinding and Rebinding

In general, JNDI binds, unbinds and rebinds are expensive operations. However, these operations become a bigger bottleneck in clustered environments because JNDI tree changes have to be propagated to all members of a cluster. If such operations are performed too frequently, they can reduce cluster scalability significantly.

 

Performance Considerations When Running Multiple Server Instances on Multi-CPU Machines

With multi-processor machines, additional consideration must be given to the ratio of the number of available CPUs to clustered WebLogic Server instances. Because WebLogic Server has no built-in limit to the number of server instances that reside in a cluster, large, multi-processor servers, such as Sun Microsystems' Sun Enterprise 10000, can potentially host very large clusters or multiple clusters.

In order to determine the optimal ratio of CPUs to WebLogic server instances, first ensure that an application is truly CPU-bound, rather than network or disk I/O-bound. Use the following steps to determine the optional ratio of CPUs to server instances:

1. Test your application to determine the Network Requirements.

   If you discover that an application is primarily network I/O-bound, consider measures to increase network throughput before increasing the number of available CPUs. For truly network I/O-bound applications, installing a faster network interface card (NIC) may increase performance more than additional CPUs, because most CPUs would remain idle while waiting to read available sockets.

2. Test your application to determine the Disk I/O Requirements.

   If you discover that an application is primarily disk I/O-bound, consider upgrading the number of disk spindles or individual disks and controllers before allocating additional CPUs.

3. Begin performance tests using a ratio of one WebLogic Server instance for every available CPU.

4. If CPU utilization is consistently at or near 100 percent, increase the ratio of CPUs to server instances by adding an additional CPU. Add additional CPUs until utilization reaches an acceptable level. Remember, always reserve some spare CPU cycles on your production systems to perform any administration tasks that may occur.

 

---

Monitoring a WebLogic Server Domain

The following sections provide information on how to monitor WebLogic Server domains:

• Using the Administration Console to Monitor WebLogic Server

• Using JMX to Monitor WebLogic Server

• Using WLST to Monitor WebLogic Server

• Third-Party Tools to Monitor WebLogic Server

 

Using the Administration Console to Monitor WebLogic Server

The tool for monitoring the health and performance of your WebLogic Server domain is the Administration Console. See “ Monitor servers” in Administration Console Online Help.

 

Using JMX to Monitor WebLogic Server

WebLogic Server® provides its own set of MBeans that you can use to configure, monitor, and manage WebLogic Server resources. See “Developing Custom Management Utilities with JMX”.

 

Using WLST to Monitor WebLogic Server

The WebLogic Scripting Tool (WLST) is a command-line scripting interface that system administrators and operators use to monitor and manage WebLogic Server instances and domains. See “WebLogic Scripting Tool”.

 

Third-Party Tools to Monitor WebLogic Server

Oracle partners with other companies that provide production monitoring and management tools. See Production Performance Management.