Multiserver environment errors
Use this information to troubleshoot problems with setting up multiserver environments.
What kind of problem are you seeing?
- When trying to create a new profile in a mixed cell environment, a mismatch of templates can occur
- After creating and starting a cluster, the cluster does not start, and logs show that servers in the cluster are not found
- One or more nodes do not display in the administrative console
- The addNode command fails
- Application files are not present on all nodes
- In a clustered environment, a server with debug mode enabled does not start
If none of these problem solution descriptions fixes the problem:
- Browse the logs of the problem deployment manager and applications servers.
View the JVM logs.
- Look up any error messages by selecting the Reference view of the information center navigation and expanding Messages in the navigation tree.
If Java exceptions are displayed in the log files, try to determine the actual subcomponent directly involved in the problem by examining the trace stack and looking for a WAS-related class near the top of the stack (names beginning with com.ibm.websphere or com.ibm.ws) that created the exception.
For example, if the exception seems to be created by a class in the com.ibm.websphere.naming package, review the Naming services component troubleshooting tips topic.
- Ensure that all the machines in the configuration have TCP/IP connectivity to each other by running the ping command:
- From each physical server to the deployment manager
- From the deployment manager to each physical server
- Although the problem is occurring in a clustered environment, the actual cause might be indirectly related, or unrelated to clustering. Investigate all relevant possibilities:
- If an enterprise bean on one or more servers is not serving requests, review the Enterprise bean cannot be accessed from a servlet, a JSP file, a stand-alone program, or another client and Application access problems topics.
- If problems seem to occur after enabling security, review the Access problems after enabling security topic.
- If an application server stops responding to request, or spontaneously fails (its process closes), review the Web module or application server stops processing requests topic.
- If SOAP requests are not served by some servers, review the Application client SOAP request troubleshooting tips topic.
- If we have problems installing or deploying an application on servers on one or more nodes, review the Application deployment problems topic.
- If the topology consists of a Windows-based deployment manager with UNIX-based servers, browse any recently-updated .xml and .policy files on the UNIX-based platform using the vi editor to ensure that Control-M characters are not present in the files. Edit these files using the vi editor on the UNIX-based platform, to avoid inserting these characters.
- Check the steps for troubleshooting the workload management component.
- Check to see if the problem is identified and documented by looking at available online support (hints and tips, technotes, and fixes).
When trying to create a new profile in a mixed cell environment, a mismatch of templates can occur
This problem occurs because profile templates are not updated when a version 6.0.x fixpack is applied on top of version 6.0.x of WAS. To Lift restrictions on a mixed cell environment, we can run a command from the bin directory of the WAS installation root to update the profile.
For the Windows platform, issue the following command, which uses C:\IBM\WebSphere\AppServer for the default installation root:
app_server_root\bin\ws_ant.bat -buildfile updateNDProfileTemplates.xml
For UNIX and Linux platforms, issue the following commands:
- For non-AIX platforms, the default installation root is /opt/IBM/WebSphere/AppServer.
- For AIX platforms, the default installation root is /usr/IBM/WebSphere/AppServer.
- For non-AIX platforms issue the following command:
USER_INSTALL_ROOT=app_server_root/profiles/your_DM_profile_name/
- For AIX issue the following command:
USER_INSTALL_ROOT=app_server_root/profiles/your_DM_profile_name/
- export USER_INSTALL_ROOT
- app_server_root/bin/ws_ant -buildfile updateNDProfileTemplates.xml
After creating and starting a cluster, the cluster does not start, and logs show that servers in the cluster are not found
This error can occur when the configuration is not synchronized from the deployment manager to a node. If auto synchronization is enabled, wait until the synchronization has run. If we are using manual synchronization, explicitly request a synchronization to each node on the cluster.
To determine whether synchronization has occurred, look at the configuration on the node machines using the administrative console and verify that the new cluster members are defined on each node.
One or more nodes do not display in the administrative console
This problem can occur when a basic connectivity problem exists between the deployment manager server and other servers in the topology. Look for the serverindex.xml file in the deployment manager directory structure:
- If the problem node does not display in the list, review the steps for adding a node to the cluster.
- If the problem node does display in the list:
- From the deployment manager server, ping the server name as it displays in the list. If the ping command indicates no communication, verify that the host name is correct in the list, correct it if necessary, then restart the deployment manager.
- If the name that displays in the list is the short name, ping the fully qualified network name. If the corrected name works, update the list, and restart the deployment manager.
- If the problem server uses Dynamic Host Configuration Protocol (DHCP), try replacing the logical host name with the IP address and restart the deployment manager. If this action resolves the problem, be aware that you must change the serverindex.xml file each time the problem server address changes, and potentially each time the problem machine is rebooted. To avoid this problem, consider assigning a static IP address to the server.
- If we still cannot establish communication between the servers, contact the network administrator to resolve the problem, and restart the deployment manager after the problem is corrected.
The addNode command fails
This error can occur when the deployment manager Domain Name Server (DNS) configuration is set up improperly. The default installation on Linux systems uses the loopback address (127.0.0.1) as the default host address. To verify this problem, query the host name of the suspect machine. If the query returns localhost 127.0.0.1, or if the file transfer traces at the node show that the node is trying to upload files to a web address that includes 127.0.0.1, the node has an incorrect DNS configuration.
To correct this problem, update the /etc/hosts file or the name service configuration file, /etc/nsswitch.conf, to query the Domain Name Server or Network Information Server (NIS) before searching hosts.
Application files are not present on all nodes
(dist)(zos) In the WAS Network Deployment environment, application binary files are transferred to the individual nodes where applications are supported as part of the node synchronization operation. During node synchronization, application files are only propagated if their deployment descriptors specify enableDistribution=true. This flag is specified as part of the application installation procedure in the administrative console, and is stored as a property in the app_server_root/config/cells/cell_name/applications/application_name/deployment.xml file.
(iseries) In the WAS Network Deployment environment, application binary files are transferred to the individual nodes where applications are supported as part of the node sync operation. During node sync, application files are only propagated if their deployment descriptors specify enableDistribution=true. This flag is specified as part of the application installation procedure in the administrative console, and is stored as a property in the profile_root/config/cells/cell_name/applications/application_name/deployment.xml file.
To confirm this problem, check to see whether the enableDistribution flag is set. If it is already set to true, ensure that the target node is configured to run auto file synchronization.
If both of these settings are correct and the problem persists, manually perform a synchronization. If the application files still do not display in the installation directory, use the EARExpander tool in app_server_root/bin directory to expand the EAR file from the repository to the installation destination. On remote nodes, the repository displays in the config/cells/cell_name/applications/application_name.ear/ directory.
In a clustered environment, a server with debug mode enabled does not start
This problem occurs when the following three conditions exist:
- Multiple server processes are configured to run on the same node
- More than one server has debug mode enabled
- The debug arguments for more than one of the servers are left at the default values, so that more than one server in the node is trying to use the same debug port (port number 7777).
The server does not start because multiple servers processes running on the same physical host machine with debug enabled cannot use the same debug port.
To correct this problem, for each server:
- On the administrative console, click Servers > Server Types > WebSphere application servers > server_name > Java and Process Management > Process definition > JVM.
- Update the debug argument so that the address of the debug port (address=port number) is unique for each server process.
Related tasks
Troubleshooting administration
Workload is not getting distributed Workload management component troubleshooting tips