Troubleshooting tips

To help you identify and resolve problems, the product has a unified logging component. We can also use the IBM Support Assistant Data Collector (ISADC) command tool, in ${wlp_install.dir}/bin, to collect diagnostic files, such as log files, configuration files or to run traces.

If you receive an exception message, information about the message is available in Messages.

The Java API documentation for each Liberty profile API is detailed in the APIs section of the information center, and is also available as a separate .zip file in one of the javadoc subdirectories of...

${wlp.install.dir}/dev

• Runtime environment known restrictions.
• Developer Tools known restrictions.

For details of the main known restrictions that apply when using the Liberty profile, see Runtime environment known restrictions.

Check that the JDKs are at a supported level

If you experience problems that are not readily explained, check that the JDKs we are using are compliant with Java v6 or later, and are at a current service level. See: Minimum supported Java levels.

A deadlock can occur when using Oracle based JVMs using Java v6. If we are using an affected JVM or JDK, the following settings can help prevent the deadlock from occurring:

• Enable the following VM option:
  -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass

• Set the Equinox framework option to use classname locking for classloading by setting the following Equinox configuration option:
  -Dosgi.classloader.lock=classname

These can be set in a Java properties file, for example jvm.options, when starting the Liberty server.

Troubleshooting security

This section describes some common security problems and solutions we can choose.

SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:LdapRegistry/cn=steven,o=myCompany,c=US.

This error happens when an unauthenticated user tries to access a session created by an authenticated user. Session security, enabled by default, prevents unauthorized access of the sessions. Only the user who created a session can access it.

This error can happen when we use a JSP (login.jsp for example) for the form-login and the SSO token sent by the browser is expired. Because the SSO token is expired, the user is prompted to log in again using the login.jsp page configured for the form-login. The jsp page, by default, tries to get a session. This session was originally created by the user whose token is expired. However, the token is expired and the user is not authenticated, no credentials are established when accessing this session that results in this error.

To avoid this error, restart the browser that starts a new session, or configure the login.jsp file to not create the session by default. If we choose to update the login.jsp file, set...

<%@ page session="false" %>

CWWKS9104A: Authorization failed for user {0} while invoking {1} on {2}. The user is not granted access to any of the required roles: {3}.

This error occurs when we do not have authorization to the roles required by the application. Make sure that the user or the group it belongs to is mapped to at least one of the roles that are mentioned in the error message. A user-to-role mapping defined in...

ibm-application-bnd.xmi/xml

...takes precedence over a mapping defined in server.xml. Check both resources to ensure that the correct mapping is defined.

CWWKZ0013E: It is not possible to start two applications called {0} followed by unexpected security behavior and error messages such as CWWKS9104A.

This error occurs when we specify the application in both the server.xml using theapplication element and in the dropins folder. As a result, the application is attempted to be installed twice and the security configuration in server.xml might or might not take effect. To fix this, we must remove your application from the dropins folder and copy it to another directory. If we have to leave it in the dropins folder, we must disable the application monitoring using the following code in server.xml:

<applicationMonitor dropinsEnabled="false"/>

An unauthenticated user was able to access my servlet or JSP.

A user with a principal of UNAUTHENTICATED (or the unauthenticated SAF user on z/OS ) is created when authentication fails or when the servlet or JSP is unprotected. An unauthenticated user can access the servlet or JSP if we do not define any security constraints or if you map the EVERYONE special subject to the role required by the application. Review the user-to-role mappings in the ibm-application-bnd.xmi/xml and server.xml files. Take one of the following options:

• If the servlet or JSP is unprotected, add security constraints to the deployment descriptor of the application.

• If we do not want any unauthenticated user to access the application, remove the EVERYONE special subject from the mapping for that role.

• If a certain user cannot be authorized to the servlet or JSP, review who is mapped to that role by examining the ibm-application-bnd.xmi/xml and server.xml files. We can map a role to a user, group, or special subject. If the role is mapped to the EVERYONE special subject, any user is granted access. If the role is mapped to the ALL_AUTHENTICATED special subject, any authenticated user is granted access. Remove these special subjects to further limit who can access the servlet or JSP. Finally, check what group the user belongs to. Although the user might not have explicit access, the group might be mapped to the role. In this case, remove the user from the authorized group or remove the group from role mapping.

Single sign-on (SSO) does not work.

If SSO does not work, verify the different Liberty profile servers that use the same LTPA keys, password, and ssoCookieName attribute of webAppSecurity element have the same Universal Time (UTC) and share the same user registry. Also, if the token expires or if the cookie is sent from a web browser after changing the user registry in a manner that is inconsistent, such as modifying the realm or removing the user the cookie represents, the SSO fails and the user is prompted to enter the credential information again. See: Customize SSO configuration using LTPA cookies

Debugging security public APIs.

WSSecurityHelper and RegistryHelper are loaded even if security is not enabled, for example, if a security feature - appSecurity-1.0, appSecurity-2.0 or zosSecurity-1.0 - is not specified. If security is not enabled, then WSSecurityHelper.isServerSecurityEnabled() and WSSecurityHelper.isGlobalSecurityEnabled() methods both return false, and RegistryHelper.getUserRegistry() method returns null.

Other security public API classes might not be loaded if security is not enabled. If we try to access these classes and call a method on one of these classes, we might get a java.lang.NoClassDefFoundError exception.

To avoid getting java.lang.NoClassDefFoundError exceptions, we must first test to see whether security is enabled by calling the WSSecurityHelper.isServerSecurityEnabled() or WSSecurityHelper.isGlobalSecurityEnabled() class, and then call other security public API classes only when security is enabled. See Security public APIs for examples of this coding technique.

This behavior is different from the full profile. In full profile, all classes are always loaded regardless of whether security is enabled or not.

Cannot authenticate users with Unicode characters

To authenticate users whose names contain Unicode characters, set the character encoding type used by the Liberty server to UTF-8 by adding the following jvm option to the server start command:

-Dclient.encoding.override=UTF-8

As well as specifying the charset and pageEncoding in your login page. Here is an example for specifying these parameters on a login JSP page:

<%@page contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>

java.lang.annotation.AnnotationFormatError: java.lang.IllegalArgumentException:Wrong type at constant pool index at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:87)

This error can occur when an OpenID Connect or OAuth provider uses a Database Store for client registration with some JDK 7 versions.

To avoid this problem, upgrade to JDK version 7.1.

Troubleshooting LDAP

This section describes some common LDAP problems and solutions we can choose.

FFDC1015I: An FFDC Incident has been created: javax.naming.ServiceUnavailableException: myldapserver.mycompany.com:636; socket closed com.ibm.ws.security.registry.ldap.internal.LdapRegistry 298

This message in messages.log indicates that the configured LDAP server cannot be reached. Check the LDAP server to verify that it is running and that its IP address can be accessed from the Liberty profile server.

The javax.naming.CommunicationException: simple bind failed: myldapserver.mycompany.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]

If we enable SSL on the LDAP server without copying the signer of the LDAP server into the truststore referenced in the LDAPSSLSettings element, a connection with the LDAP server fails with an SSL handshake error. Make sure that you copy the signer of the LDAP server into the truststore.

The javax.naming.CommunicationException: myldapserver.mycompany.com:389 [Root exception is java.net.BindException: Address already in use: connect]

This message might appear in the FFDC logs and indicates that the usable sockets on the local server are exhausted, which can result in a failure when connecting to the LDAP server. Make sure that the socket is not used and try again.

CWWKS1100A: Authentication did not succeed for user ID xxxxx. An invalid user ID or password was specified

This FFDC exception might happen on the server even though the user mentioned in the message is a valid user on the LDAP server under heavy load. With the LDAP configuration, we can add the reuseConnection=false property or disable it using the developer tools. To fix the problem, set this property to the default value of true.

Troubleshooting SSL

This section describes some common SSL problems and solutions we can choose.

CWWKS9105E: Could not determine the SSL port for automatic redirection.

This error occurs when we try to access an application that redirects to an SSL port and the SSL port is not available. The port might not be available because of a missing SSL configuration or some error in the SSL configuration definition. Check the SSL configuration in the server.xml file to make sure that it exists and is correct. See Secure communications with the Liberty profile.

FFDC1015I: An FFDC Incident has been created: "java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field com.ibm.ws.config.internal.cm.ManagedServiceFactoryTracker aSyncReadNupdate. Exception thrown while trying to read configuration and update ManagedServiceFactory. Exception = java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field" at ffdc_12.04.18_16.09.42.0.log

This error occurs when a keystore element exists in the configuration without an ID field. If we use a minimal SSL configuration, set the ID field to defaultKeyStore.

We might get an exception if using a LDAP user registry with sslEnabled and a sslRef value is not specified.

For example, a configuration has sslEnabled set to true but there is not a sslRef value, as shown in the following example:

<ltldapRegistry id="ldap" realm="SampleLdapIDSRealm" 
host="ccwin12.austin.ibm.com" port="636" ignoreCase="true"
baseDN="o=ibm,c=us"
bindDN="cn=root"
bindPassword="rootpwd"
ldapType="IBM Tivoli Directory Server"
idsFilters="ibm_dir_server"
sslEnabled="true"
searchTimeout="8m" />

You must enter a sslRef value. If we are using a minimal SSL configuration that is similar to the following:

<ltkeyStore id="defaultKeyStore" location="key.jks"
password="mypassword" />

the sslRef field should be set to defaultSSLConfig.

If a custom SSL configuration is configured, the name of that configuration should be placed in the sslRef field.

If we use a JDK from the WebSphere Application Server, we might see the following error if SSL is enabled on the Liberty Server.

java.net.SocketException: java.lang.ClassNotFoundException: Cannot find the specified class com.ibm.websphere.ssl.protocol.SSLSocketFactory 
      at javax.net.ssl.DefaultSSLSocketFactory.a(SSLSocketFactory.java:11) 
      at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:6) 
      at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:161) 
      at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:36) 
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1184) 
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:390) 
      at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:75) 
      at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.loadJMXServerInfo(RESTMBeanServerConnection.java:142) 
      at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.<init>(RESTMBeanServerConnection.java:114) 
      at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:315) 
      at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:103) 

This error occurs because the WebSphere Application Server SSL socket factories are not supported by the Liberty profile. You can get past this problem by editing the java.security file from the JDK and comment out the two lines that set the SSL socket factories to the WebSphere Application Server SSL socket factories. The following example shows the two lines in the java.security file that must be commented out:

#ssl.SocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLSocketFactory
#ssl.ServerSocketFactory.provider=com.ibm.websphere.ssl.protocol.SSLServerSocketFactory

Troubleshooting CORBA/IIOP

This section describes some common CORBA problems and solutions we can choose.

If we use a JDK from the WebSphere Application Server, we might see the following error if the application uses CORBA/IIOP communications.

15:21:58.096 com.ibm.rmi.pi.InterceptorManager runPreInit:178 Init Process ORBRas [default]  java.lang.ClassNotFoundException:
com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI
        at com.ibm.CORBA.iiop.UtilDelegateImpl.loadClass(UtilDelegateImpl.java:685)
        at javax.rmi.CORBA.Util.loadClass(Util.java:246)
        at com.ibm.rmi.pi.InterceptorManager.runPreInit(InterceptorManager.java:172)
        at com.ibm.rmi.corba.ORB.initializeInterceptors(ORB.java:664)
        at com.ibm.CORBA.iiop.ORB.initializeInterceptors(ORB.java:1084)
        at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1359)
        at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1271)
        at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1694)
        at org.omg.CORBA.ORB.init(ORB.java:371) 
        ...

This error occurs because the WebSphere Application Server ORB Intercepters are not supported by the liberty profile. We can get past this problem by editing the orb.properties file from the JDK (which is usually found in WebSphere's <JAVA_HOME>/jre/lib directory, though the user may have overridden it with their own copy in <USER_HOME>) to not use these interceptors. The following example shows the lines in the orb.properties file that must be commented out:

# WS Interceptors
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.Transaction.JTS.TxInterceptorInitializer
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ejs.ras.RasContextSupport
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.ClientRIWrapper
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.activity.remote.cos.ActivityServiceClientInterceptor
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI 
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.debug.olt.ivbtrjrt.OLT_RI
#org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.wlm.client.WLMClientInitializer

# WS ORB & Plugins properties #com.ibm.ws.orb.transport.ConnectionInterceptorName=com.ibm.ISecurityLocalObjectBaseL13Impl.SecurityConnectionInterceptor

Troubleshooting logging and tracing

This section describes some common problems with logging and tracing.

The java.util.logging -- Java logging programming interface.

The Liberty profile does not support using the logging.properties file to configure java.util.logging. Use java code, for example in a deployed application or user feature, to create and add java.util.logging handlers, filters, or formatters.

Since the Liberty profile server manages the java.util.logging logger levels in accordance with the traceSpecification attribute of the logging configuration element, we should avoid using the Logger.setLevel method.

Apply fix packs and interim fixes to an archive install

If we installed your Liberty profile runtime environment from an archive file, rather than using Installation Manager, we must take special measures when you apply service updates.

See Apply a fix pack to a Liberty profile archive installation and Apply an interim fix to a Liberty profile archive installation.

Troubleshooting performance

This section describes some common performance problems and solutions we can choose.

High CPU usage by the application monitor.

This error can occur if your application monitor has many files under the apps/ directory and is polling too frequently.

To avoid this problem there are a number of things we can change.

1. Increase the value of the pollingRate attribute.

2. Update the server.xml to include an applicationMonitor element with an updateTrigger that is not polled.

   server.xml                                                                      
   <applicationMonitor updateTrigger="mbean" /> 

3. Reduce the number of files under the apps/ directory.

For more information about the applicationMonitor element, see Control dynamic updates.

Subtopics

• Logging and Trace
  
• Timed operations and JDBC calls
  
• Binary logging
  
• Runtime environment known restrictions
  
• Developer Tools known restrictions
  
• Messages
  

Tasks:

1. Use the Liberty profile as an application development environment
2. Enable basic authentication for web services access
3. Enable client certificate authentication for web services access
4. Secure web services at the transport level
5. Enable SSL communication for web services access
6. Developer Tools known restrictions