Troubleshooting the proxy server

This topic helps you to solve problems that you might encounter with the proxy server.

(zos) Proxy server errors are logged in the joblog, Consult the following list if you are having problems with the proxy server.

Proxy server errors are logged in the SystemOut.log, proxy.log, or local.log files. Consult the following list if you are having problems with the proxy server.

This topic references one or more of the application server log files. As a recommended alternative, we can configure the server to use the High Performance Extensible Logging (HPEL) log and trace infrastructure instead of using SystemOut.log , SystemErr.log, trace.log, and activity.log files on distributed and IBM i systems. We can also use HPEL in conjunction with the native z/OS logging facilities. If we are using HPEL, we can access all of the log and trace information using the LogViewer command-line tool from the server profile bin directory. See the information about using HPEL to troubleshoot applications for more information on using HPEL.

• The proxy server was created successfully, but I am unable to start it.
  Check the SYSOUT file for port conflicts. Use the netstat a command to see if any of the endpoints associated with the proxy server are already being used. We can find the ports in the console by clicking > Servers > Proxy servers > <server_name> > Ports.

  If the proxy server fails to start when attempting to start it as a non-privleged user on UNIX systems, check for the following message in the logs:

  ChannelFramew E   CHFW0029E: Error initializing chain HTTPS_PROXY_CHAIN because of  exception  com.ibm.wsspi.channel.framework.exception.RetryableChannelException: Permission  denied
  TCPPort       E   TCPC0003E: TCP Channel TCP_7 initialization failed.  The socket  bind failed for host * and port 80.  The port may already be in use.

  Change the ports of the PROXY_HTTP_ADDRESS and PROXY_HTTPS_ADDRESS transport chains to values greater than 1024.

• The proxy server routes requests to the web container over an administration port.
  The proxy server is located in front of several web containers. The configuration requires that the web containers listen to the non-default ports such as 9061, 9081, and so on. This scenario is the default case when multiple application servers are on the same machine, which forces new and different ports to be used in the configuration. In this scenario, the proxy server might route an application request to the web container over the administration port of 9061, instead of using the expected port of 9081.

  Add the listening port numbers of the web container to the virtual host that is associated with the target application. This process will ensure that the proxy server routes the request to the web container over the correct port number.

• The proxy server started, but I am unable to access the application resources through the endpoints for the proxy server.
  Ensure that the endpoints for the proxy server are among the host aliases in the virtual host associated with the application.

• The proxy server routes to another core group.
  Verify that core group bridges exist between the core groups in the cell, and that the processes that are chosen to be bridges are restarted. If there is a firewall between the core group, verify that the correct ports are open for core group bridge traffic.

• The proxy server is unable to route requests to another cell.
  Review the core group bridge settings. If the HMGR0149E error message is logged on any server, usually on a server acting as a core group bridge, then the security settings for the cell need to be modified to allow communication.

  See the topic entitled Export Lightweight Third Party Authentication keys for more information about configuring cell security between multiple cells.

• Receiving a blank page when making a request to the proxy.
  Consider the following actions:

  • Update the virtual host. Ensure that the target application and routing rule are assigned to a virtual host that includes the proxy server listening ports (default: HTTP 80, HTTPS 443). Add the proxy server listening ports to the application, or routing rule virtual host, or use the proxy_host virtual host.

  • Stop the conflicting process. Check the system to ensure that no other process (for example, Apache, IBM HTTP Server, and so on) is running that uses the proxy server ports (default: HTTP 80, HTTPS 443). If this problem occurs, the proxy server seems to start normally, but is unable to receive requests on the affected listening port. Check the system as follows:

    1. Stop the proxy server.

    2. Query the system using netstat and ps commands to determine if an offending process is using the port on which the proxy server is listening.

    3. If an offending process is found, stop the process and configure your system so that the process is not started during system startup.

  • Enable proxy routing. Ensure that proxy routing is enabled for the web module of the application. Proxy routing is enabled by default, so if no proxy properties are modified, disregard this solution. Otherwise, see Customize routing to applications for instructions on modifying the proxy properties.

  • Test direct request. Ensure that the target application is installed by making a request directly to the application server. If a response is not received, then the problem is with the application server and not the proxy server. Verify this case by going through the proxy server after we can receive a response directly from the application server.

• HTTP 404 (File not found) error received from the proxy server. Consider the following actions:

  • Update the virtual host. Ensure that the target application and routing rule are assigned to a virtual host that includes the proxy server listening ports (default: HTTP 80, HTTPS 443). Add the proxy server listening ports to the application, or routing rule virtual host, or use the proxy_host virtual host.

  • Enable proxy routing. Ensure that proxy routing is enabled for the web module of the application. Proxy routing is enabled by default, so if no proxy properties are modified, disregard this solution. Otherwise, see Customize routing to applications for instructions on modifying the proxy properties.

  • Test direct request. Ensure that the target application is installed by making a request directly to the application server. If a response is not received, then the problem is with the application server and not the proxy server. Verify this case by going through the proxy server when we can receive a response directly from the application server.

• Unable to make SSL requests to application or routing rule. Ensure that the virtual host of the application or routing rule includes a host alias for the proxy server SSL port (default: 443).

• Unable to connect to the proxy server...request times out. Stop the conflicting process. Check the system to ensure that no other process (for example, Apache, IBM HTTP Server, and so on) is running that uses the proxy server ports (default: HTTP 80, HTTPS 443). If this situation occurs, the proxy server seems to start normally, but is unable to receive requests on the affected listening port. Check the system, as follows:

  1. Stop the proxy server.

  2. Query the system using netstat and ps commands to determine if an offending process is using a port on which the proxy server is listening.

  3. If an offending process is found, stop the process and configure your system so that the process is not started during system startup.

• Did not receive a response from the error page application when the HTTP error occurred (for example, 404). Ensure that the error page URI is entered correctly. Also, verify the Handle remote errors option is selected if you are handling HTTP error responses from back-end servers. For more detailed information, refer to Overview of the custom error page policy and the custom error page policy section of Proxy server settings.

• What packages do I enable when tracing the proxy server?
  All of the following packages are not needed for every trace, but if unsure, use all of them:

  • *=info

  • WebSphere Proxy=all

  • GenericBNF=all

  • HAManager=all

  • HTTPChannel=all

  • TCPChannel=all

  • WLM*=all

  • DCS=all

  • ChannelFrameworkService=all

  • com.ibm.ws.dwlm.*=all

  • com.ibm.ws.odc.*=all

• How do I enable SSL on/off load?
  SSL on/off load is referred to as the transport protocol in the console, and transport protocol is a web module property. Refer to Customize routing to applications to see how to configure web module properties. No SSL on/off load or transport protocol properties exist for routing rules because the transport protocol is inherent to the generic server cluster specified in the routing rule.

• When fronted by IBM HTTP Server or a web server plug-in, how do I configure the proxy server so I do not have to add a port for it to the virtual host?
  For the proxy server to trust the security-related information included in a request, such as private headers that the product adds, add the originator of the request to the proxy server trusted security proxies list. For example, add an IBM HTTP Server or a web server plug-in that sends requests to the proxy server, to the proxy server trusted security proxies list. The web server plug-in can then send product added private header information that, contains the virtual host information of a request. If the proxy does not trust these private headers from the web server plug-in, or from any client, the proxy server adds its own private headers, which requires the addition of proxy server HTTP and HTTPS ports to the virtual host. Typically, when a web server plug-in is used with the proxy server, the intent is to use the proxy server as a back-end server. Therefore, add the plug-in as a trusted security proxy to avoid having to expose the proxy server ports. Route requests from a plug-in to a proxy server provides more information about configuring a web server plug-in to use with the proxy server. Proxy server settings provides more information about setting up trusted security proxies.

• The proxy server seems to hang under stress, or Too Many Files Open exceptions display in ffdc or SystemErr.log.
  Under high connection loads, the number of file system descriptors might become exhausted and the proxy server may seem to hang and drop Too Many Files Open exceptions in the ffdc directory or in the SystemError.log file because it is unable to open a socket. To alleviate this problem, modify one or more of the following parameters at the operating system level, and at the proxy server level to optimize the use of connections for the proxy server:

  • Operating system tuning for Windows 2003, and XP

    • TcpTimedWaitDelay - Determines the time that must elapse before TCP/IP releases a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP releases closed connections faster and can provide more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, or an adjustment because of a low throughput caused by multiple connections in the TIME_WAIT state.

      View or set this value as follows:

      1. Use the regedit command and access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey to create a new REG_DWORD value named TcpTimedWaitDelay.

      2. Set the value to decimal 30, which is Hex 0x0000001e. This value sets the wait time to 30 seconds.

      3. Stop and restart the system.

      Information	Value
      Default value	0xF0, which sets the wait time to 240 seconds (4 minutes).
      Recommended value	A minimum value of 0x1E, which sets the wait time to 30 seconds.

    • MaxUserPort - Determines the highest port number that TCP/IP can assign when an application requests an available user port from the system. View or set this value as follows:

      1. Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named MaxUserPort.

      2. Set this value to at least decimal 32768.

      3. Stop and restart the system.

      Information	Value
      Default value	None
      Recommended value	At least decimal 32768.

  • Operating system tuning for Linux

    • timeout_timewait parameter - Determines the time that must elapse before TCP/IP releases a closed connection and can reuse its resources. This interval between closure and release is known as the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT state. View or set this value by issuing the following command to set the timeout_timewait parameter to 30 seconds:
      echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

    • Linux file descriptors (ulimit) - Number of open files that are supported. The default setting is typically sufficient for most applications. If the value set for this parameter is too low, a file open error, memory allocation failure, or connection establishment error might display.

      View or set this value by checking the UNIX reference pages on the ulimit command for the syntax of different shells. Set the ulimit command to 65535 for the KornShell shell (ksh), by issuing the ulimit -n 65535 command. Use the ulimit -a command to display the current values for all limitations on system resources.

      Information	Value
      Default value	1024
      Recommended value	65535

  • Operating system tuning for AIX

    • TCP_TIMEWAIT - Determines the time that must elapse before TCP/IP releases a closed connection and can reuse its resources. This interval between closure and release is known as the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter, if the running application requires rapid release or the creation of new connections, or if a low throughput occurs due to many connections sitting in the TIME_WAIT state. View or set this value by issuing the following command to set the TCP_TIMEWAIT state to 15 seconds:
      /usr/sbin/no -o tcp_timewait =1

    • AIX file descriptors (ulimit) - Number of open files that are permitted. The default setting is typically sufficient for most applications. If the value set for this parameter is too low, errors can occur when opening files or establishing connections, and a memory allocation error might display. To prevent WebSphere Application Server from running short on resources, remove the upper limits (ulimit) for resources on the user account on which the WAS process runs.

      View or set this value by changing the ulimit settings as follows:

      1. Open the command window.

      2. Type smitty users to open the AIX configuration program.

      3. Select Change or Show Characteristics of a user.

      4. Type the name of the user account on which the WAS runs.

      5. Press Enter.

      6. Change the following settings to the indicated value:

         Information	Value
         Soft FILE Size	-1
         Soft CPU Time	-1
         Soft STACK Size	-1
         Soft CORE File Size	-1
         Hard FILE Size	-1
         Hard CPU Time	-1
         Hard STACK Size	-1
         Hard CORE File Size	-1

      7. Press Enter to save changes.

      8. Log out and log into the account.

      9. Restart the product.

      Information	Value
      Default value	2000
      Recommended value	unlimited

  • Operating system tuning for HP-UX

    HP-UX nfile kernel parameter - Maximum number of files that can be open on the system at any given time. Not specifying a large enough number might restrict the system processing capacity.

    HP-UX ninode kernel parameter - Maximum number of open inodes that can be in memory. An open inode is associated with each unique open file. Therefore, the value specified for the ninode parameter should be larger than the number of unique files that must be open at the same time.

  • Operating system tuning for Solaris

    • Solaris file descriptors (ulimit) - Number of open files that are supported. If the value specified for this parameter is too low, a file open error, memory allocation failure, or connection establishment error might display. View or set this value by checking the UNIX reference pages on the ulimit command for the syntax of different shells.

      • Issue the ulimit -a command to display the current values for all limitations on system resources.

      • Issue the ulimit -n 65535 command to set the ulimit command to 65535 for the KornShell shell (ksh).

      • The maximum number of files that can be open for a process is also affected by the rlim_fd_max, and rlim_fd_cur settings for the global kernel limit. We might need to increase the values of these settings in the /etc/system file.

      Solaris 10 provides a new mechanism for setting kernel parameters. Issue one of the following commands to specify a new limit for the max number of file descriptor kernel parameter on Solaris 10:

      1. To change the value for the current shell session:
         prctl -n process.max-file-descriptor -r -v 65535 $$

      2. To make the change a system wide change:

         projmod -sK 'process.max-file-descriptor=(privileged,65535,deny)'
         system 

         Caution should be used when issuing this command because this command changes the setting for all users, and all projects.

      3. To change the value for all projects that are owned by user root:

         projmod -sK 'process.max-file-descriptor=(privileged,65535,deny)'
         user.root

      4. To change the value for all projects owned by non-root user

         projmod -sK 'process.max-file-descriptor=(privileged,65535,deny)'
         user.username

    • TCP_TIME_WAIT_INTERVAL - Notifies TCP/IP on how long to keep the connection control blocks closed. After the applications complete the TCP/IP connection, the control blocks are kept for the specified time. When high connection rates occur, a large backlog of the TCP/IP connections accumulate and can slow server performance. The server can stall during certain peak periods. If the server stalls, the netstat command shows that many of the sockets that are opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2 state. Visible delays can occur for up to four minutes, during which time the server does not send any responses, but CPU utilization stays high with all of the activities in system processes.

      View or set this value using the get command to determine the current interval and the set command to specify an interval of 30 seconds. For example:

      ndd -get /dev/tcp tcp_time_wait_interval ndd -set /dev/tcp tcp_time_wait_interval 30000

      Information	Value
      Default value	240000 milliseconds, which is equal to 4 minutes.
      Recommended value	60000 milliseconds

  • Proxy server tuning

    • Persistent requests - A persistent request is one that is sent over an existing TCP connection. We can maximize performance by increasing the number of requests that are received over a TCP connection from a client. The value should represent the maximum number of embedded objects, for instance GIF and so on, in a web page +1.

      View or set this value in the WAS console by clicking Servers > Proxy Servers > server_name > Proxy server transports > HTTP_PROXY_CHAIN/HTTPS_PROXY_CHAIN.

      Information	Value
      Default value	100
      Recommended value	A value that represents the maximum number of embedded objects in a web page + 1.

    • Outbound connection pool size - The proxy server pools outbound connections to target servers and the number of connections that resides in the pool is configurable. If the connection pool is depleted or empty, the proxy server creates a new connection to the target server. Under high concurrent loads, increase the connection pool size should to a value of the expected concurrent client load to achieve optimal performance.

      View or set this value in the WAS console by clicking Servers > Proxy Servers > server_name > Proxy server transports > HTTP Proxy Server Settings. In the Content Server Connection section, increase the maximum connections per server field to a value that is equal to or greater than the expected maximum number of connected clients. Save the changes, synchronize the changes to the proxy server node, and restart the proxy server.

      Information	Value
      Recommended value	Value consistent to the expected concurrent client load.

    • Outbound request time-out - Often times, the back-end application servers that are fronted by the proxy server may be under high load and may not respond in an adequate amount of time, therefore the connections on the proxy server may be tied up from waiting for the back-end application server to respond. Alleviate this by configuring the amount of time the proxy server waits for a response from the target server. This is the Outbound Request Time-out value. By managing the amount of time the proxy server waits for a slow back-end application server, connections are freed up faster and used for other request work.

      View or set this value in the console by clicking Servers > Server Types > WebSphere proxy servers > proxy_server_name > HTTP Proxy Server Settings. In the Content Server Connection section, set Outbound Request Time-out to a value that represents the acceptable response time from the point of view of the client.

      Information	Value
      Default value	120
      Recommended value	A value that represents the acceptable response time from the point of view of the client.

Subtopics

• Troubleshooting request routing and workload management through the proxy server
  This section provides information for how to troubleshoot request traffic that flows through the proxy server.

Related tasks

Proxy server setup

Export Lightweight Third Party Authentication keys

Use High Performance Extensible Logging to troubleshoot applications