Web server plug-in troubleshooting tips

If we are having problems using a web server plug-in, try these steps:

• Review...
  plugins_root/logs/web_server/http_plugin.log

  Look up any error or warning messages in the message table.

• Review the web server's error and access logs to see if the web server is having a problem:

  IBM HTTP Server and Apache	access.log and error.log.
  Domino Web Server	httpd-log and httpd-error.
  Sun Java System	access and error.
  Microsoft IIS	timedatestamp.log.

If these files don't reveal the cause of the problem, follow these additional steps.

Plug-in Problem Determination Steps

The plug-in provides very readable tracing which can be beneficial in helping to figure out the problem. By setting the LogLevel attribute in the config/plugin-cfg.xml file to Trace, we can follow the request processing to see what is going wrong.

(HPUX) Note: If we are using a Veritas File System with large file support enabled, file sizes up to two terabytes are allowed. In this case, if we set the LogLevel attribute in the plugin-cfg.xml file to LogLevel=Trace, then the http_plugin.log file might grow quickly and consume all available space on the file system. Therefore, we should set the value of the LogLevel attribute to ERROR or DEBUG to prevent high CPU utilization..

At a high level, complete these steps.

1. The plug-in gets a request.
2. The plug-in checks the routes defined in the plugin-cfg.xml file.
3. It finds the server group.
4. It finds the server.
5. It picks the transport protocol, HTTP or HTTPS.
6. It sends the request.
7. It reads the response.
8. It writes it back to the client.

We can see this very clearly by reading through the trace for a single request:

• Determine if the plug-in has successfully loaded into the web server.

  • Check to make sure the http_plugin.log file has been created.

  • If it has, look in it to see if any error messages indicate some sort of failure that took place during plug-in initialization. If no errors are found look for the following stanza, which indicates that the plug-in started normally. Ensure that the timestamps for the messages correspond to the time you started the web server:

    [Thu Jul 11 10:59:15 2002] 0000009e 000000b1 - PLUGIN: ------------System Information---------- 
    [Thu Jul 11 10:59:15 2002] 0000009e 000000b1 - PLUGIN: Bld date: Jul  3 2002, 15:35:09 
    [Thu Jul 11 10:59:15 2002] 0000009e 000000b1 - PLUGIN: Web server: IIS 
    [Thu Jul 11 10:59:15 2002] 0000009e 000000b1 - PLUGIN: Hostname = SWEETTJ05 
    [Thu Jul 11 10:59:15 2002] 0000009e 000000b1 - PLUGIN: OS version 4.0, build 1381, 'Service Pack 6' 
    [Thu Jul 11 10:59:15 2002] 0000009e 000000b1 - PLUGIN: -------------------------------------------- 
    

  • Some common errors are:

    lib_security: loadSecurityLibrary: Failed to load gsk library

    Either GSKit did not get installed or the wrong version of GSKit got installed. To determine which situation occurred:

    • (Windows) On a Windows platform, search for the file gsk7ssl.dll
    • (UNIX) On a UNIX platform, search for a libgsk7*.so files in the /usr/lib directory.

    If we cannot find the appropriate file, try reinstalling the plug-in with the correct GSKit version to see if this fixes the problem.

    ws_transport: transportInitializeSecurity: Keyring wasn't set

    The HTTPS transport defined in the configuration file was prematurely terminated and did not contain the Property definitions for the keyring and stashfile. Check your XML syntax for the line number given in the error messages that follow this one to make sure the Transport element contains definitions for the keyring and stashfiles before it is terminated.

  • If the http_plugin.log file is not created, check the web server error log to see if any plug-in related error messages have been logged there that indicate why the plug-in is failing to load. Typical causes of this can include failing to correctly configure the plug-in with the web server environment. Check the documentation for configuring the web server that we are using with the web server plug-in.

• Determine whether there are network connection problems with the plug-in and the various application servers defined in the configuration. Typically we will see the following message when this is the case:

  ws_common: websphereGetStream: Failed to connect to app server, OS err=%d

  Where %d is an OS specific error code related to why the connect() call failed. This can happen for a variety of reasons.

  • Ping the machines to make sure they are properly connected to the network. If the machines cannot be pinged, there is no way for the plug-in to contact them. Possible reasons for this problem include:
    • Firewall policies that limit the traffic from the plug-in to the application server.

    • The machines are not on the same network.

  • If we are able to ping the machines then the probable cause of the problem is that the port is not active. The port might not be active because the application server or cluster is not started or the application server has gone down for some reason. To verify that this is the problem, we can try to manually telnet into the port that the connect is failing on. If we cannot telnet into the port the application server is not up and that problem needs to be resolved before the plug-in can successfully connect.

• Determine whether other activity on the machines where the servers are installed is impairing the ability of the server to service a request. Check the processor utilization as measured by the task manager, processor ID, or some other outside tool to see if it:
  • Is not what was expected.
  • Is erratic rather than a constant.
  • Shows that a newly added member of the cluster is not being utilized.
  • Shows that a failing member that has been fixed is not being utilized.

• Check the administrative console for server status.

  Check the administrative console to ensure that the application servers are started. View the administrative console for error messages or look in the JVM logs.

  (ZOS) Check the administrative console to ensure that the application servers are started. View the administrative console for error messages.

• In the administrative console, select the problem application server and view its installed applications to verify that they are started.

If none of these steps solves the problem:

• For specific problems that can cause web pages and their contents not to display, see Web resource (JSP file, servlet, html file, image, etc) will not display in the information center.

• Check to see if the problem has been identified and documented using the links in Diagnose and fix problems: Resources for learning.

• If we do not see a problem that resembles ours, or if the information provided does not solve our problem, contact IBM support for further assistance.

Change in behavior when the web server uses an insecure transport

If we have secure and insecure transports defined, and a secure transport cannot be obtained due to a system failure, the web server plug-in uses an insecure transport. This is the default behavior.

This behavior has been changed in WAS v8.5.5. If a system failure occurs when attempting a secure connection, and there is an insecure transport, the web server plug-in does not use that transport. The administrator is made aware of the problem and can fix it using secure connections.

There are 3 options available to the administrator to address this problem:

1. Correct the SSL issue so that the HTTPS transport is available (this is the recommended option).

2. Remove the HTTPS transports if SSL is not used, and regenerate the plug-in.

3. Set UseInsecure=true in the plugin-cfg.xml file to revert back to the previous default behavior.

To revert to the previous default behavior, we can enable it by setting a custom property on the administrative console. Select...

webserver-><webserver>->plugin properties->custom properties

...and set UseInsecure to true.

(ZOS) If the HTTP server is not a managed server (it does not show up in the administrative console), we must manually edit the plugin-cfg.xml file. Read the Installing and configuring the plug-in for V5.3 HTTP Server for z/OS topic for more information.

To revert to the previous default behavior, we can enable it by setting a custom property on the administrative console. Select...

webserver-><webserver>->plugin properties->custom properties

...and set UseInsecure to true.

How to fix GSK_ERROR_BAD_CERT (gsk rc=414) in plug-in log

Implement a web server plug-in