Administration and administrative console troubleshooting
As a recommended alternative, we can configure the server to use the High Performance Extensible Logging (HPEL) log and trace infrastructure SystemErr.log, trace.log, and activity.log files on distributed and IBM i systems. We view HPEL log and trace information using the logViewer.
In WebSphere Application Server products, administrative functions are supported by:
- The application server (such as server1) process
- The dmgr process in the WAS ND product
The process must be running to use the administrative console. The wsadmin command-line utility has a local mode we can use to perform some administrative functions, even when the server process is not running.
What kind of problem are you seeing?
- Server status and messages in the console view are not current
- Role-based authorization fails
- When starting or stopping a server using a wsadmin interactive scripting session, we receive an exception indicating read timed out
- Problems starting or using the administrative console or wsadmin utility
- The AdminConfig.parents method cannot be used to obtain the valid parent types of some configuration types
Server status and messages in the console view are not current
When connecting to an Application Server that uses a Simple Object Access Protocol (SOAP) connection for a long time, the following problems begin to occur:
- Under the status column in the Servers view on an administrative console panel, the status of the server does not refresh.
- Server messages are not updated in the administrative console.
- A decrease of system resources occurs as numerous ports are created and left in the TIME_WAIT state.
This problem persists even once you restart the server or you start another server that uses the SOAP connection port.
The problem occurs because the SOAP connector does not support connection pooling. If the Application Server has many ongoing operations that use the SOAP connector, the Application Server quickly opens and closes many ports. Due to the nature of the underlying TCP/IP protocol, these ports remain in the TIME_WAIT state for some time prior to the operating system can reclaim them. The number of ports that WAS opens can exceed the limit that the operating system imposes. Under this condition, the opening of additional ports fails through the SOAP connector until the operating system reclaims ports.
Use the following options to work around the problem:
- Increase the operating system limits on the number of ports.
- For Rational Application Developer, the wsadmin utility, or Java applications that use the Java Management Extension (JMX) connectors, switch to the RMI connector.
- Wait until few or no ports are in the TIME_WAIT state prior to performing new operations through Rational Application Developer or the administrative console.
Role-based authorization fails
When we make a Java Management Extension (JMX) call such as getAttribute, setAttribute, invoke, and so on in your application, the caller requires an administrative role with sufficient permissions. The required role depends on the MBean attribute or method that the JMX caller calls and can be one of administrator, configurator, monitor, or operator. If one of the administrative roles is not assigned to the caller, or if the role is assigned, but the caller does not have the required permissions, the application receives a role-based authorization failure, for example:
SECJ0305I: Role based authorization check failed for securityname server.domain.name:3890/user.id, accessId user:server.domain.name:3890/uid=user.id,ou=xxxx,dc=yyy,dc=zzz while invoking method getNodeName on resource Server and module Server.If the caller of the application cannot be assigned one of the administrative roles, the application can log in with one of the roles on behalf of the caller. For example:try { // Create a LoginContext to authenticate a user ID and password. javax.security.auth.login.LoginContext lc = new javax.security.auth.login.LoginContext("WSLogin", new com.ibm.websphere.security.auth.callback.WSCallbackHandlerImpl("adminuser", "adminpassword")); // perform the login lc.login(); // Get the authenticated subject. javax.security.auth.Subject adminSubject = lc.getSubject(); // Define the action that will take place using the authenticated Subject // We can define this action anywhere in the code, the action // is reference in the WSSubject.doAs that follows. java.security.PrivilegedAction adminAction = new java.security.PrivilegedAction() { public Object run() { try { // Get the WebSphere AdminService. AdminService adminservice = AdminServiceFactory.getAdminService(); // Get the WebSphere Admin Local Server MBean instance. ObjectName objectname = adminservice.getLocalServer(); // Get the Node name. String nodeName = (String)adminservice.getAttribute(objectname, "nodeName"); // Get the Application Server name. String serverName = (String)adminservice.getAttribute(objectname, "name"); // Get the Application Server Process ID. String serverPid = (String)adminservice.getAttribute(objectname, "pid"); // Return a result, for this example, just return the serverPid. return serverPid; } catch (Exception e) { e.printStackTrace(); } return null; } }); // Invoke an AdminClient resource using the authenticated subject. // This example demonstrates the action of creating an // administrative client and returning a String value to use outside // the doAs block. String myData = (String) com.ibm.websphere.security.auth.WSSubject.doAs(adminSubject, adminAction); // Use "myData" later on.... } catch (javax.security.auth.login.LoginException e) { e.printStackTrace(); }
When starting or stopping a server using a wsadmin interactive scripting session, we receive an exception indicating read timed out
When starting or stopping a server using a wsadmin interactive scripting session, we receive an exception indicating read timed out, for example:
WASX7015E: Exception running command: "$AdminControl startServer server1 Node1;" exception information: com.ibm.websphere.management.exception.ConnectorException org.apache.soap.SOAPException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out; targetException=java.net.SocketTimeoutException: Read timed out]This exception occurs because the timeout value is too small. Increase the timeout value specified by the com.ibm.SOAP.requestTimeout property in the soap.client.props file in the profile_root/properties directory for a single server edition or in the profile_root/properties directory for a network deployment installation. The value we choose depends on a number of factors such as the size and the number of the applications installed on the server, the speed of our machine, and the level of usage of our machine. The default value of the com.ibm.SOAP.requestTimeout property is 180 seconds.
(iSeries) This exception occurs because the timeout value is too small. To fix this, increase the timeout value specified by the com.ibm.SOAP.requestTimeout property in the soap.client.props file in the profile_root/properties directory for a single server edition or in the profile_root/properties directory for a network deployment installation. The value we should choose depends on a number of factors such as the size and the number of the applications installed on the server, the speed of our machine, and the level of usage of our machine. The default value of the com.ibm.SOAP.requestTimeout property is 180 seconds.
Problems starting or using the administrative console or wsadmin utility
If we have problems starting or using the administrative console or wsadmin utility, verify that the supporting server process is started and that it is healthy.
- For the application server process, refer to the following files:
- profile_root/logs/server/startServer.log for the message that indicates that the server started successfully: ADMU3000I: Server server1 open for e-business; process id is nnnn..
- (iSeries) profile_root/logs/server/startServer.log for the message that indicates that the server started successfully: ADMU3000I: Server server1 open for e-business; process id is nnnn..
- profile_root/logs/server/SystemOut.log
- (iSeries) profile_root/logs/server/SystemOut.log
- The server log files for the message that indicates that the server started successfully: WSVR0001I: Server server open for e-business.
- For the WAS ND product, refer to the following files:
- profile_root/logs/dmgr/startServer.log for the message that indicates that the server started successfully: ADMU3000I: Server dmgr open for e-business; process id is nnnn.
- (iSeries) profile_root/logs/dmgr/startServer.log for the message that indicates that the server started successfully: ADMU3000I: Server dmgr open for e-business; process id is nnnn.
- profile_root/logs/dmgr/SystemOut.log for the message that indicates that the server started successfully: ADMU3000I: Server dmgr open for e-business; process id is nnnn.
- (iSeries) profile_root/logs/dmgr/SystemOut.log for the message that indicates that the server started successfully: ADMU3000I: Server dmgr open for e-business; process id is nnnn.
- The server log files for this message indicating that the server started successfully:WSVR0001I: Server dmgr open for e-business.
- For the z/OS product, check the job output.
- Look up any error messages in these files in the message reference table. Select the Reference view in the information center navigation, and click Messages. A message like WASX7213I: This scripting client is not connected to a server process when trying to start wsadmin indicates that either the server process is not running, the host machine where it is running is not accessible, or that the port or server name that the wsadmin utility uses is incorrect.
- Verify that we are using the right port number to communicate with the administrative console or the wsadmin server:
Look in the SystemOut.log file.
(ZOS) Look in the joblogs file.
- The line ADMC0013I: SOAP connector available at port nnnn indicates the port that the server is using to listen for wsadmin functions.
- The com.ibm.ws.scripting.port property in the profile_root/properties/wsadmin.properties file controls the port used by the wsadmin utility to send requests to the server.
- (iSeries) The com.ibm.ws.scripting.port property in the profile_root/properties/wsadmin.properties file controls the port used by the wsadmin utility to send requests to the server.
- If port value is different from the value shown in the SystemOut.log file, either change the port number in the wsadmin.properties file, or specify the correct port number when starting the wsadmin utility using the -port port_number property on the command line.
The com.ibm.ws.scripting.port property in the profile_root/properties/wsadmin.properties file controls the port used by the wsadmin utility to send requests to the server.
- (iSeries) If the port value is different from the value shown in the SystemOut.log file, either change the port number in the wsadmin.properties file, or specify the correct port number when starting the wsadmin utility using the -port port_number property on the command line.
The com.ibm.ws.scripting.port property in the profile_root/properties/wsadmin.properties file controls the port used by the wsadmin utility to send requests to the server.
- (iSeries) If the port value is different from the value shown in the server log files, either change the port number in the wsadmin.properties file, or specify the correct port number when starting the wsadmin utility using the -port port_number property on the command line.
The message SRVE0171I: Transport http is listening on port nnnn (default 9060) indicates the port that the server uses to listen for administrative console requests.
- If the port value is different than the one specified in the web address for the administrative console, change the web address in the browser to the correct value. The default is http://localhost:9060/ibm/console.
- Use the telnet command to test that the host name where the application server or deployment manager is running, is reachable from the system where the browser or wsadmin program is used. If we can ping the host name, no firewall or connectivity issues exist.
- If the host where the application server or deployment manager is running is remote to the machine from which the client browser or wsadmin command is running, ensure that the appropriate host name parameter is correct. Verify:
- The host name in the browser web address for the console.
- The -host host name option of the wsadmin command used to direct the wsadmin utility to the correct server.
- (iSeries) WAS technical support might ask you to trace the administrative component for detailed problem determination. The trace specification for this component is com.ibm.websphere.management.*=all=enabled:com.ibm.ws.management.*=all=enabled"
The AdminConfig.parents method cannot be used to obtain the valid parent types of some configuration types
When using the AdminConfig.parents method to obtain the allowed parent types for a specific configuration type, for example, "JavaVirtualMachine", an informational message may be returned instead of the desired value.
wsadmin>print AdminConfig.parents( "JavaVirtualMachine")The following informational message is returned:WASX7351I: The parents command cannot be used to find the parents of type "JavaVirtualMachine"The following function demonstrates one possible work around for the AdminConfig.parents() method that deals with the issue previously mentioned:
#--------------------------------------------------------------------- # Name: parentTypes() # Role: To search for configuration types that may contain objects of # the specified type should the result of calling # AdminConfig.parents( Type ) # be: # WASX7351I: The parents command cannot be used to find the parents of type ... # Note: Unlike the AdminConfig.list() command, this routine will not # raise an InvalidConfigDataTypeException for an unknow datatype #--------------------------------------------------------------------- def parentTypes( Type ) : import os WSAStypes = AdminConfig.types().splitlines() if Type in WSAStypes : result = AdminConfig.parents( Type ) if result.startswith( 'WASX7351I' ) : result = [] for thisType in WSAStypes : if AdminConfig.attributes( thisType ).find( Type ) > -1 : result.append( thisType ) result = (os.linesep).join( result ) else : print 'parentTypes error: unknown / unrecognized type:', Type result = None return resultIf none of these steps solves the problem, see if the specific problem we are having is addressed in the Installation completes but the administrative console does not start topic. Check to see if the problem has been identified and documented using the links in the Diagnose and fix problems: Resources for learning topic. If we do not see a problem that resembles ours, or if the information provided does not solve your problem, contact IBM support for further assistance.
(iSeries) If none of these steps solves the problem, see if the specific problem we are having is addressed in the Installation completes but the administrative console does not start topic. Check to see if the problem is identified and documented using the links in the Diagnose and fix problems: Resources for learning topic.
For current information available from IBM Support on known problems and their resolution, see the following topics on the IBM support page:
IBM Support has documents that can save you time gathering the information needed to resolve this problem. Prior to opening a PMR, see the following topics on information gathering on the IBM support page:
Troubleshoot administration