Troubleshooting the proxy server

 

+

Search Tips   |   Advanced Search

 

About this task

Consult the following list if you are having problems with your proxy server:

 

Procedure

• The proxy server panels do not display in the administrative console. Ensure that you have the WAS V6.0.2 update installed on the deployment manager and that the deployment manager profile is augmented using the proxy_augment template, as described in Install and setup the proxy server.

• The proxy server was created successfully, but I am unable to start it. Check the SYSOUT file for port conflicts. Use the netstat -aa command to see if any of the endpoints that are associated with the proxy server are already being used. We can find the ports in the administrative console by clicking...

  Servers | Proxy servers | <server_name> | Ports

  If the proxy server fails to start when attempting to start it as a non-privleged user on UNIX systems, check for the following message in the logs:

  ChannelFramew E   CHFW0029E: Error initializing chain HTTPS_PROXY_CHAIN because of 
  exception 
  com.ibm.wsspi.channel.framework.exception.RetryableChannelException: Permission 
  denied
  TCPPort       E   TCPC0003E: TCP Channel TCP_7 initialization failed.  The socket 
  bind failed for host * and port 80.  The port may already be in use.
  

  Change the ports of the PROXY_HTTP_ADDRESS and PROXY_HTTPS_ADDRESS transport chains to values greater than 1024.

• The proxy server started, but I am unable to access the application resources through the endpoints for the proxy server. Ensure that the endpoints for the proxy server are among the host aliases in the virtual host that are associated with the application.

• The proxy server routes to another core group. Verify that core group bridges exist between the core groups in the cell, and that the processes that are chosen to be bridges are restarted. If there is a firewall between the core group, verify that the correct ports are open for core group bridge traffic.

• The proxy server routes to another cell. Review the core group bridge settings. Verify that the peer access point group names match in each cell. Check the peer ports against the bridge interfaces to verify that they are correct. If bridge interfaces or peer ports are added or changed, restart all bridge interfaces.

• Receiving a blank page when making a request to the proxy. Consider the following actions:

  • Update the virtual host. Ensure that the target application and routing rule are assigned to a virtual host that includes the proxy server listening ports (default: HTTP 80, HTTPS 443). Add the proxy server listening ports to the application, or routing rule virtual host, or use the proxy_host virtual host.

  • Stop the conflicting process. Check your system to ensure that no other process (for example, Apache, IBM HTTP Server, and so on) is running that uses the proxy server ports (default: HTTP 80, HTTPS 443). If this problem occurs, the proxy server seems to start normally, but is unable to receive requests on the affected listening port. Check your system as follows:

    1. Stop the proxy server.

    2. Query your system using netstat and ps commands to determine if an offending process is using the port on which the proxy server is listening.

    3. If an offending process is found, stop the process and configure your system so that the process is not started during system startup.

  • Enable proxy routing. Ensure that proxy routing is enabled for the Web module of the application. Proxy routing is enabled by default, so if no proxy properties are modified, disregard this solution. Otherwise, see Customizing routing to applications for instructions on modifying the proxy properties.

  • Test direct request. Ensure that the target application is installed by making a request directly to the application server. If a response is not received, then the problem is with the application server and not the proxy server. Verify this case by going through the proxy server after we can receive a response directly from the application server.

• HTTP 404 (File not found) error received from the proxy server. Consider the following actions:

  • Update the virtual host. Ensure that the target application and routing rule are assigned to a virtual host that includes the proxy server listening ports (default: HTTP 80, HTTPS 443). Add the proxy server listening ports to the application, or routing rule virtual host, or use the proxy_host virtual host.

  • Enable proxy routing. Ensure that proxy routing is enabled for the Web module of the application. Proxy routing is enabled by default, so if no proxy properties are modified, disregard this solution. Otherwise, see Customizing routing to applications for instructions on modifying the proxy properties.

  • Test direct request. Ensure that the target application is installed by making a request directly to the application server. If a response is not received, then the problem is with the application server and not the proxy server. Verify this case by going through the proxy server when we can receive a response directly from the application server.

• Unable to make Secure Sockets Layer (SSL) requests to application or routing rule. Ensure that the virtual host of the application or routing rule includes a host alias for the proxy server SSL port (default: 443).

• Unable to connect to the proxy server...request times out. Stop the conflicting process. Check your system to ensure that no other process (for example, Apache, IBM HTTP Server, and so on) is running that uses the proxy server ports (default: HTTP 80, HTTPS 443). If this situation occurs, the proxy server seems to start normally, but is unable to receive requests on the affected listening port. Check your system, as follows:

  1. Stop the proxy server.

  2. Query your system using netstat and ps commands to determine if an offending process is using a port on which the proxy server is listening.

  3. If an offending process is found, stop the process and configure your system so that the process is not started during system startup.

• Did not receive a response from the error page application when the HTTP error occurred (for example, 404). Ensure that the error page URI is entered correctly. Also, make sure that the Handle remote errors option is selected if you are handling HTTP error responses from back-end servers. For more detailed information, refer to Overview of the custom error page policy and the custom error page policy section of Proxy server settings .

• What packages do I enable when tracing the proxy server? All of the following packages are not needed for every trace, but if unsure, use all of them:

  • *=info

  • WebSphere Proxy=all

  • GenericBNF=all

  • HAManager=all

  • HTTPChannel=all

  • TCPChannel=all

  • WLM*=all

  • DCS=all

  • ChannelFrameworkService=all

  • com.ibm.ws.dwlm.*=all

  • com.ibm.ws.odc.*=all

• How do I enable SSL on/off load? SSL on/off load is referred to as the transport protocol in the administrative console, and transport protocol is a Web module property. Refer to Customizing routing to applications to see how to configure Web module properties. No SSL on/off load or transport protocol properties exist for routing rules because the transport protocol is inherent to the generic server cluster that is specified in the routing rule.

• When fronted by IBM HTTP Server or a plug-in, how do I configure the proxy server so I do not have to add a port for it to the virtual host? For the proxy server to trust the security-related information, for example WAS private headers, of a request, add the originator of the request to the proxy server trusted security proxies list. For example, add an IBM HTTP Server or a plug-in sending requests to the proxy server to the proxy server trusted security proxies list. The plug-in sends WebSphere Application Server private header information that among other things, contains the virtual host information of a request. If the proxy does not trust the WAS private headers from the plug-in (or any client), the proxy server adds its own WAS private headers, which requires the addition of proxy server ports (HTTP and HTTPS) to the virtual host. Most likely, when using the plug-in with the proxy server, the intent is to use the proxy server as a back-end server. Be sure to add the WAS plug-in as a trusted security proxy to avoid having to expose the proxy server ports. Refer to Routing requests from a plug-in to a proxy server for more information about configuring the WebSphere Application Server plug-in to use with the proxy server. Refer to Proxy server settings for more information about trusted security proxies.

• The proxy server seems to "hang" under stress, or "Too Many Files Open" exceptions display in ffdc or SystemErr.log. Under high connection loads, the number of file system descriptors might become exhausted and the proxy server may seem to hang and drop "Too Many Files Open" exceptions in the ffdc directory or in the SystemError.log file because it is unable to open a socket. The problem can be alleviated by setting certain tuning parameters at the operating system level and at the proxy server level that optimize the use of connections for the proxy server:

  • Operating system tuning for Windows 2000, 2003, and XP

    • TcpTimedWaitDelay - Determines the time that must elapse before TCP/IP releases a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP releases closed connections faster and can provide more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, or an adjustment because of a low throughput caused by multiple connections in the TIME_WAIT state.

      View or set this value as follows:

      1. Use the regedit command and access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey to create a new REG_DWORD value named TcpTimedWaitDelay.

      2. Set the value to decimal 30, which is Hex 0x0000001e. This value sets the wait time to 30 seconds.

      3. Stop and restart your system.

      Default value	0xF0, which sets the wait time to 240 seconds (4 minutes).
      Recommended value	A minimum value of 0x1E, which sets the wait time to 30 seconds.

    • MaxUserPort - Determines the highest port number that TCP/IP can assign when an application requests an available user port from the system. View or set this value as follows:

      1. Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named MaxUserPort.

      2. Set this value to at least decimal 32768.

      3. Stop and restart your system.

      Default value	None
      Recommended value	At least decimal 32768.

  • Operating system tuning for Linux

    • timeout_timewait parameter - Determines the time that must elapse before TCP/IP releases a closed connection and can reuse its resources. This interval between closure and release is known as the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, and a low throughput due to many connections sitting in the TIME_WAIT state.

      View or set this value by issuing the following command to set the timeout_timewait parameter to 30 seconds:

      echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout           

    • Linux file descriptors (ulimit) - Specifies the number of open files that are supported. The default setting is typically sufficient for most applications. If the value set for this parameter is too low, a file open error, memory allocation failure, or connection establishment error might display.

      View or set this value by checking the UNIX reference pages on the ulimit command for the syntax of different shells. Set the ulimit command to 65535 for the KornShell shell (ksh), by issuing the ulimit -n 65535 command. Use the ulimit -a command to display the current values for all limitations on system resources.

      Default value	1024
      Recommended value	65535

  • Operating system tuning for AIX

    • TCP_TIMEWAIT - Determines the time that must elapse before TCP/IP releases a closed connection and can reuse its resources. This interval between closure and release is known as the TIME_WAIT state, or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Adjust this parameter, if the running application requires rapid release or the creation of new connections, or if a low throughput occurs due to many connections sitting in the TIME_WAIT state.

      View or set this value by issuing the following command to set the TCP_TIMEWAIT state to 15 seconds:

      /usr/sbin/no -o tcp_timewait =1          

    • AIX file descriptors (ulimit) - Specifies the number of open files that are permitted. The default setting is typically sufficient for most applications. If the value set for this parameter is too low, errors can occur when opening files or establishing connections, and a memory allocation error might display. To prevent WAS from running short on resources, remove the upper limits (ulimit) for resources on the user account on which the WebSphere Application Server process runs.

      View or set this value by changing the ulimit settings as follows:

      1. Open the command window.

      2. Type smitty users to open the AIX configuration program.

      3. Select Change or Show Characteristics of a user.

      4. Type the name of the user account on which the WebSphere Application Server runs.

      5. Press Enter.

      6. Change the following settings to the indicated value:

         Soft FILE Size	-1
         Soft CPU Time	-1
         Soft STACK Size	-1
         Soft CORE File Size	-1
         Hard FILE Size	-1
         Hard CPU Time	-1
         Hard STACK Size	-1
         Hard CORE File Size	-1

      7. Press Enter to save changes.

      8. Log out and log into your account.

      9. Restart WebSphere Application Server

      Default value	2000
      Recommended value	unlimited

  • Operating system tuning for Solaris

    • TCP_TIME_WAIT_INTERVAL - Notifies TCP/IP on how long to keep the connection control blocks closed. After the applications complete the TCP/IP connection, the control blocks are kept for the specified time. When high connection rates occur, a large backlog of the TCP/IP connections accumulate and can slow server performance. The server can stall during certain peak periods. If the server stalls, the netstat command shows that many of the sockets that are opened to the HTTP server are in the CLOSE_WAIT or FIN_WAIT_2 state. Visible delays can occur for up to four minutes, during which time the server does not send any responses, but CPU utilization stays high with all of the activities in system processes.

      View or set this value by using the get command to determine the current interval and the set command to specify an interval of 30 seconds. For example:

      ndd -get /dev/tcp tcp_time_wait_interval     
      ndd -set /dev/tcp tcp_time_wait_interval 30000  

      Default value	240000 milliseconds, which is equal to 4 minutes.
      Recommended value	60000 milliseconds

  • Proxy server tuning

    • Persistent requests - A persistent request is one that is sent over an existing TCP connection. We can maximize performance by increasing the number of requests that are received over a TCP connection from a client. The value should represent the maximum number of embedded objects, for instance GIF and so on, in a Web page +1.

      View or set this value in the WebSphere Application Server administrative console by clicking Servers > Proxy Servers > server_name > Proxy server transports > HTTP_PROXY_CHAIN/HTTPS_PROXY_CHAIN

      Default value	100
      Recommended value	A value that represents the maximum number of embedded objects in a Web page + 1.

    • Outbound connection pool size - The proxy server pools outbound connections to target servers and the number of connections that resides in the pool is configurable. If the connection pool is depleted or empty, the proxy server creates a new connection to the target server. Under high concurrent loads, increase the connection pool size should to a value of the expected concurrent client load to achieve optimal performance.

      View or set this value in the WAS administrative console by clicking...

      Servers | Proxy Servers | server_name | HTTP Proxy Server Settings

      In the Content Server Connection section, increase the maximum connections per server field to a value that is equal to or greater than the expected maximum number of connected clients. Save your changes, synchronize the changes to the proxy server node, and restart the proxy server.

      Recommended value

      Value consistent to the expected concurrent client load.

    • Outbound request time-out - Often times, the back-end application servers that are fronted by the proxy server may be under high load and may not respond in an adequate amount of time, therefore the connections on the proxy server may be tied up from waiting for the back-end application server to respond. Alleviate this by configuring the amount of time the proxy server waits for a response from the target server. This is the Outbound Request Time-out value. By managing the amount of time the proxy server waits for a slow back-end application server, connections are freed up faster and used for other request work.

      View or set this value in the WAS administrative console by clicking Servers > Proxy Servers server_name > HTTP Proxy Server Settings. In the Content Server Connection section, set Outbound Request Time-out to a value that represents the acceptable response time from the point of view of the client.

      Default value	120
      Recommended value	A value that represents the acceptable response time from the point of view of the client.

Related information

Setting up the proxy server

 

---