Troubleshooting tips for Liberty

The troubleshooting information describes solutions for common problems.

To help you identify and resolve problems, the product has a unified logging component. For more information, see Logging and Trace.

If we receive an exception message, information about the message is available in Messages.

The Java API documentation for each Liberty API is detailed in the Program interfaces (Javadoc) section of the online IBM documentation, and is also available as a separate .zip file in one of the javadoc subdirectories of the ${wlp.install.dir}/dev directory. Details of the main known restrictions that apply when we use Liberty are provided in the following two topics:

Check that your JDKs are at a supported level

If we experience problems that are not readily explained, check that the JDKs we are using are compliant with Java Version 7 or later, and are at a current service level.

See Minimum supported Java levels.

Troubleshooting security

This section describes some common security problems and solutions we can choose.

SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:LdapRegistry/cn=steven,o=myCompany,c=US.

This error happens when an unauthenticated user tries to access a session created by an authenticated user. Session security that is enabled by default prevents unauthorized access of the sessions. Only the user who created a session can access it.

See session security.

This error can happen when we use a JSP (login.jsp for example) for your form-login and the SSO token sent by the browser is expired. Because the SSO token is expired, the user is prompted to log in again using the login.jsp page configured for the form-login. The jsp page, by default, tries to get a session. This session was originally created by the user whose token is expired. Because the token is expired and the user is not authenticated, no credentials are established when you access this session, which results in this error.

To avoid this error, restart the browser that starts a new session, or configure the login.jsp file to not create the session by default. If we choose to update the login.jsp file, set <%@ page session="false" %>.

CWWKS9104A: Authorization failed for user {0} while invoking {1} on {2}. The user is not granted access to any of the required roles: {3}.

This error occurs when we do not have authorization to the roles required by the application. Make sure the user or the group it belongs to is mapped to at least one of the roles that are mentioned in the error message. A user-to-role mapping defined in the ibm-application-bnd.xmi/xml file takes precedence over a mapping that is defined in the server.xml file. Check both resources to ensure that the correct mapping is defined.

See Configure authorization for applications in Liberty.

CWWKS9104A: Authorization failed for user {0}.

This error can occur if we specify both an application and webApplication for the same context root. If a conflict happens the latest configuration defined is ignored and causes an unexpected error, such as CWWKS9104A.

CWWKZ0013E: It is not possible to start two applications called {0} followed by unexpected security behavior and error messages such as CWWKS9104A.

This error occurs when we specify the application in both the server.xml using the application element and in the dropins folder. As a result, the application is attempted to be installed twice and the security configuration in the server.xml file might or might not take effect. To fix this problem, we must remove the application from the dropins folder and copy it to another directory. If we must leave it in the dropins folder, we must disable the application monitoring using the following code in the server.xml file:

<applicationMonitor dropinsEnabled="false"/>

An unauthenticated user was able to access my servlet or JSP.

A user with a principal of UNAUTHENTICATED (or the unauthenticated SAF user on z/OS) is created when authentication fails or when your servlet or JSP is unprotected. An unauthenticated user can access your servlet or JSP if we do not define any security constraints or if you map the EVERYONE special subject to the role required by the application. Review the user-to-role mappings in the ibm-application-bnd.xmi/xml and server.xml files. Take one of the following options:

If wer servlet or JSP is unprotected, add security constraints to the deployment descriptor of the application.
See Authentication.
If we do not want any unauthenticated user to access the application, remove the EVERYONE special subject from the mapping for that role. For more information, see Configure authorization for applications in Liberty.
If a certain user cannot be authorized to your servlet or JSP, review who is mapped to that role by examining the ibm-application-bnd.xmi/xml and server.xml files. We can map a role to a user, group, or special subject. If your role is mapped to the EVERYONE special subject, any user is granted access. If wer role is mapped to the ALL_AUTHENTICATED special subject, any authenticated user is granted access. Remove these special subjects to further limit who can access your servlet or JSP. Finally, check what group the user belongs to. Although the user might not have explicit access, the group might be mapped to the role. In this case, remove the user from the authorized group or remove the group from role mapping.
See Configure authorization for applications in Liberty.

Single sign-on (SSO) does not work.

If SSO does not work, make sure that the different Liberty servers that use the same LTPA keys, password, and ssoCookieName attribute of webAppSecurity element have the same Universal Time (UTC) and share the user registry. Also, if the token expires or if the cookie is sent from a web browser after we change the user registry in a manner that is inconsistent, such as modifying the realm or removing the user that the cookie represents, the SSO fails, and the user is prompted to enter the credential information again.

See Customizing SSO configuration using LTPA cookies in Liberty.

Debugging security public APIs.

WSSecurityHelper and RegistryHelper are loaded even if security is not enabled, for example, if a security feature - appSecurity-1.0, appSecurity or zosSecurity-1.0 - is not specified. If security is not enabled, then WSSecurityHelper.isServerSecurityEnabled() and WSSecurityHelper.isGlobalSecurityEnabled() methods both return false, and RegistryHelper.getUserRegistry() method returns null.

Other security public API classes might not be loaded if security is not enabled. If we try to access these classes and call a method on one of these classes, we might get a java.lang.NoClassDefFoundError exception.

To avoid getting java.lang.NoClassDefFoundError exceptions, we must first test to see whether security is enabled by calling the WSSecurityHelper.isServerSecurityEnabled() or WSSecurityHelper.isGlobalSecurityEnabled() class, and then call other security public API classes only when security is enabled. For examples of this coding technique, see Security public APIs.

Note that this behavior is different from the WebSphere Application Server traditional. In WebSphere Application Server traditional, all classes are always loaded regardless of whether security is enabled or not.

Cannot authenticate users with Unicode characters

To authenticate users whose names contain Unicode characters, set the character encoding type used by the Liberty server to UTF-8 by adding the following jvm option to the server start command:

-Dclient.encoding.override=UTF-8

We must also specify the charset and pageEncoding in your login page. Here is an example for specifying these parameters on a login JSP page:

<%@page contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>

java.lang.annotation.AnnotationFormatError: java.lang.IllegalArgumentException:Wrong type at constant pool index at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:87)

This error can occur when an OpenID Connect or OAuth provider uses a Database Store for client registration with some JDK 7 versions.

To avoid this problem, upgrade to JDK version 7.1.

Troubleshooting LDAP

This section describes some common LDAP problems and solutions we can choose.

FFDC1015I: An FFDC Incident has been created: javax.naming.ServiceUnavailableException: myldapserver.mycompany.com:636; socket closed com.ibm.ws.security.registry.ldap.internal.LdapRegistry 298

This message in messages.log indicates that the configured LDAP server cannot be reached. Check the LDAP server to verify that it is running and that its IP address can be accessed from the Liberty server.

The javax.naming.CommunicationException: simple bind failed: myldapserver.mycompany.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]

If we enable SSL on the LDAP server without copying the signer of the LDAP server into the truststore that is referenced in the LDAPSSLSets element, a connection with the LDAP server fails with an SSL handshake error. Make sure that you copy the signer of the LDAP server into your truststore.

The javax.naming.CommunicationException: myldapserver.mycompany.com:389 [Root exception is java.net.BindException: Address already in use: connect]

This message might appear in the FFDC logs and indicates that the usable sockets on the local server are exhausted, which can result in a failure when you connect to the LDAP server. Make sure that the socket is not used and try again.

CWWKS1100A: Authentication did not succeed for user ID xxxxx. An invalid user ID or password was specified

This FFDC exception might happen on the server even though the user who is mentioned in the message is a valid user on the LDAP server under heavy load. With the LDAP configuration, we can add the reuseConnection=false property or disable it using the developer tools. To fix the problem, set this property to the default value of true.

Login, authentication, or authorization is slow for LDAP or a federated repository.

Several reasons can exist for a slow login. Review the following list for some common causes.

If the LDAP configuration for referrals is set to follow, then any referrals to other LDAP instances are followed on LDAP requests. This situation can result in contacting one or more additional LDAP servers, depending on the LDAP configuration, and thus, additional time is spent. For example, the LDAP1 server could follow a referral to the LDAP2 server, which could follow a referral to the LDAP3 server. If we do not need to acquire more information from other LDAP servers, set referrals to ignore. Additionally, one LDAP server could be experiencing a problem and error that might not display in the JVM logs. Examples include a TCP read timeout or a DNS issue when the LDAP server talks to a referred LDAP server. To diagnose these situations, we can capture packets to see how many calls are being made and if any delays or errors exist due to an LDAP server that is following a referral.
A firewall or other software closes connections to LDAP. By default, a pool of LDAP connections is maintained to improve performance. If a cached connection is closed remotely, a new connection is made and put back in the context pool. This process can cause a delay and can cause errors to be logged in the JVM logs. To avoid this situation, set the context pool timeout to less than the remote connection closure time. For example, if a firewall closes the connection at 10 minutes, you could set the connection pool timeout for 9 minutes. Instead of encountering a failed connection and creating a new connection, the expiration is checked on a connection from the pool and a new connection is created, skipping the failure step. Example errors include the following one:
For more information on setting the context pool timeout, see the contextPool element on the ldapRegistry element.
With federated repositories, all repositories and registries are checked to ensure that a unique user is in the realm. If a repository or registry is responding slowly, every call is slow even if the user is not in the slow registry. Ensure that all participating base entries are responding promptly.

Occasional connection exceptions accessing the LDAP server. For example, java.net.SocketException: Connection reset

A firewall or other software closes connections to LDAP. By default, a pool of LDAP connections is maintained to improve performance. If a cached connection is closed remotely, a new connection is made and put back in the context pool. This process can cause a delay and can cause errors to be logged in the JVM logs. To avoid this situation, set the context pool timeout to less than the remote connection closure time. For example, if a firewall closes the connection at 10 minutes, the connection pool timeout could be set for 9 minutes. Instead of encountering a failed connection and creating a new connection, the expiration is checked on a connection from the pool and a new connection is created, skipping the failure step.

Troubleshooting SSL

This section describes some common SSL problems and solutions we can choose.

This error occurs when our Liberty environment attempts to communicate with an external server using SSL and a specific hostname, but the server returns an SSL certificate that does not contain the hostname. For more information on troubleshooting, see Troubleshooting SSL and TLS on the Open Liberty website.

CWWKS9105E: Could not determine the SSL port for automatic redirection.

This error occurs when you try to access an application that redirects to an SSL port and the SSL port is not available. The port might not be available because of a missing SSL configuration or some error in the SSL configuration definition. Check the SSL configuration in the server.xml file to make sure that it exists and is correct. For more information, see Secure communications with Liberty.

FFDC1015I: An FFDC Incident has been created: java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field com.ibm.ws.config.internal.cm.ManagedServiceFactoryTracker aSyncReadNupdate. Exception thrown while trying to read configuration and update ManagedServiceFactory. Exception = java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field at ffdc_12.04.18_16.09.42.0.log

This error occurs when a keystore element exists in the configuration without an ID field. If we use a minimal SSL configuration, set the ID field to defaultKeyStore.

We might get an exception if we use an LDAP user registry with sslEnabled and a sslRef value is not specified.

For example, a configuration has sslEnabled set to true but there is not a sslRef value, as shown in the following example:

<ltldapRegistry id="ldap" realm="SampleLdapIDSRealm" 
host="ccwin12.austin.ibm.com" port="636" ignoreCase="true"
baseDN="o=ibm,c=us"
bindDN="cn=root"
bindPassword="rootpwd"
ldapType="IBM Tivoli Directory Server"
idsFilters="ibm_dir_server"
sslEnabled="true"
searchTimeout="8m" />

You must enter a sslRef value. If we are using a minimal SSL configuration that is similar to the following configuration, the sslRef field must be set to defaultSSLConfig. If we are using the PKCS12 keystore, your minimal configuration would be similar to the following one.

<ltkeyStore id="defaultKeyStore" location="key.p12"
password="mypassword" />

If we are using the JKS keystore, your minimal configuration would be similar to the following one.

<ltkeyStore id="defaultKeyStore" location="key.jks"
password="mypassword" />

If a custom SSL configuration is configured, the name of that configuration must be placed in the sslRef field.

If we use a JDK from the WebSphere Application Server, we might see the following error if SSL is enabled on our Liberty Server.

java.net.SocketException: java.lang.ClassNotFoundException: Cannot find the specified class com.ibm.websphere.ssl.protocol.SSLSocketFactory 
      at javax.net.ssl.DefaultSSLSocketFactory.a(SSLSocketFactory.java:11) 
      at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:6) 
      at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:161) 
      at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:36) 
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1184) 
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:390) 
      at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:75) 
      at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.loadJMXServerInfo(RESTMBeanServerConnection.java:142) 
      at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.<init>(RESTMBeanServerConnection.java:114) 
      at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:315) 
      at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:103)

This error occurs because the WebSphere Application Server SSL socket factories are not supported by Liberty. We can get past this problem by taking the following steps:

Create a text file, such as my.java.security with the following two empty entries
Create a jvm.options file for our Liberty server
Add the following property to your jvm.options file, that includes the full path to your text file that you just created
To make this more reusable, we can put the my.java.security file in the server directory, and then we are able to use a relative path like this:

See Customizing the Liberty environment.

Troubleshooting CORBA/IIOP

This section describes some common CORBA problems and solutions we can choose.

If we use a JDK from the WebSphere Application Server, we might see the following error if the application uses CORBA/IIOP communications.

15:21:58.096 com.ibm.rmi.pi.InterceptorManager runPreInit:178 Init Process ORBRas [default] java.lang.ClassNotFoundException: com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI at com.ibm.CORBA.iiop.UtilDelegateImpl.loadClass(UtilDelegateImpl.java:685) at javax.rmi.CORBA.Util.loadClass(Util.java:246) at com.ibm.rmi.pi.InterceptorManager.runPreInit(InterceptorManager.java:172) at com.ibm.rmi.corba.ORB.initializeInterceptors(ORB.java:664) at com.ibm.CORBA.iiop.ORB.initializeInterceptors(ORB.java:1084) at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1359) at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1271) at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1694) at org.omg.CORBA.ORB.init(ORB.java:371) ...

This error occurs because the WebSphere Application Server Object Request Broker (ORB) interceptors are not supported by Liberty. We can resolve this problem by editing the orb.properties file from the JDK to not use these interceptors. This file is found in the WebSphere <JAVA_HOME>/jre/lib directory, though it might be overwritten with a copy in the user's <USER_HOME> directory. The following example shows the lines in the orb.properties file that must be commented out:

# WS Interceptors #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.Transaction.JTS.TxInterceptorInitializer #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ejs.ras.RasContextSupport #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.ClientRIWrapper #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.activity.remote.cos.ActivityServiceClientInterceptor #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.debug.olt.ivbtrjrt.OLT_RI #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.wlm.client.WLMClientInitializer # WS ORB & Plugins properties #com.ibm.ws.orb.transport.ConnectionInterceptorName=com.ibm.ISecurityLocalObjectBaseL13Impl.SecurityConnectionInterceptor

Troubleshooting logging and tracing

This section describes some common problems with logging and tracing.

The java.util.logging -- Java logging programming interface.: Liberty does not support using the logging.properties file to configure java.util.logging. Use Java code, for example in a deployed application or user feature to create and add java.util.logging handlers, filters, or formatters.; Since the Liberty server manages the java.util.logging logger levels in accordance with the traceSpecification attribute of the logging configuration element, avoid using the Logger.setLevel method.

Troubleshooting JAX-WS

This section describes some common JAX-WS problems and solutions we can choose.

The org.apache.cxf.bus.extension.ExtensionException occurred when you deploy web services application on Liberty.: If the application has CXF libraries and configurations that are embedded already and we want to deploy the application to Liberty, you must make sure that the jaxws-2.2 server feature is disabled by removing the feature from the server.xml file.

The java.lang.NoClassDefFoundError exception occurred when we run IBM fastpath with the Oracle JVM.

To use IBM fastpath Java Architecture for XML Binding (JAXB), we can configure the com.ibm.xml.xlxp.jaxb.opti.level custom property to control whether optimization methods are enabled for JAXB unmarshalling (deserialization) and marshalling (serialization). If we are experiencing the java.lang.NoClassDefFoundError exception when we run IBM fastpath JAXB with the Oracle JVM, we can change the value of com.ibm.xml.xlxp.jaxb.opti.level property to 0 to turn off the optimization. For example, we can use the -Dcom.ibm.xml.xlxp.jaxb.opti.level=0 property in the command line, or add the following line to the jvm.options file:

# Turn off the JAXB optimization
-Dcom.ibm.xml.xlxp.jaxb.opti.level=0

For more information on the com.ibm.xml.xlxp.jaxb.opti.level property, see the Java virtual machine custom properties page.

The java.lang.NoClassDefFoundError : com.ibm.CORBA.iiop.ORB exception occurred when we run the WebSphere Application Server traditional thin client with the Oracle JVM.

This error occurs when you try to run the WebSphere Application Server traditional thin client with the Oracle JVM and the jaxws-2.2 feature is already enabled on Liberty. To resolve this problem, we can use the -Dcom.ibm.websphere.thinclient=true property when we run the WebSphere Application Server traditional thin client as follows:

java -Dcom.ibm.websphere.thinclient=true 
     -cp <classpath_entries_including_tWAS_thinclient_jar> <WebServicesClientEntryClass> <any_more_parameters>

For similar information on the com.ibm.websphere.thinclient property, see the PM39777: ADMINISTRATIVE THIN CLIENT USING SOAP CONNECTOR AND SUN JDK DOES NOT WORK page.

An Empty file that is returned when you retrieve binary files from an MTOM service to a MTOM client.

This scenario happens when a MTOM client sent binary files successfully to an MTOM service, but an empty file is returned when you retrieve the same binary files back from the MTOM service.

As a workaround, we can create a new DataHandler and initialize the DataHandler by filling the content of the binary file. For example, use a FileDataStore object to read from I/O and return the newly created DataHandler back, and then pass the DataHandler to other web services.

...
File f = new File("received_image");
	if (f.exists()) { 		f.delete();
	}

	FileOutputStream fos = new FileOutputStream(f);
	img_in.writeTo(fos);

	FileDataSource fos_out = new FileDataSource(f);
	DataHandler img_out = new DataHandler(fos_out);
	return img_out;
...

The javax.xml.ws.soap.SOAPFaultException: Unmarshalling Error occurred when we use XMLGregorianCalendar in xsd:gMonth format with the Oracle JVM.

This error occurs when we use the XMLGregorianCalendar class to store gMonth format constants as part of the SOAP request on client side and the jaxws-2.2 feature is enabled on Liberty with the Oracle JVM. The root cause is that the Oracle JVM supports the new standard format --MM for the xsd:gMonth type while the latest JAXB RI (Reference Implementation) supports the format of --MM-- only.

To resolve this problem, we can change the Oracle JVM to the IBM JVM for both the client side and the server-side applications.

The java.io.FileNotFoundException occurred when you resolve WSDL files specified by the wsdlLocation attribute.

This error occurs when we specify the WSDL file as in the code wsdlLocation = "file:/WEB-INF/wsdl/a.wsdl" and the WSDL file is packaged within your application. To refer to the WSDL file that is packaged in the application, we must use relative URIs for the wsdlLocation attribute defined in the @WebService, @WebServiceProvider, @WebServiceClient, or @WebServieRef annotation.

The relative URI for the wsdlLocation attribute must be one of the following values:

wsdlLocation = "/WEB-INF/wsdl/a.wsdl"
wsdlLocation = "WEB-INF/wsdl/a.wsdl"

Many Create service messages occur in the messages.log file.

This scenario happens when you call the getPort or related methods in the generated client stub classes. Liberty is configured with the default logging configuration, and all the information messages are written into the messages.log file. There are lots of occurrences of messages that might be similar to the following one:

00000037 org.apache.cxf.service.factory.ReflectionServiceFactoryBean I Create Service {http://www.echo.org/}EchoService from WSDL: 
    wsjar:file:/wlp/usr/servers/default/workarea/org.eclipse.osgi/bundles/100/data/cache/
    com.ibm.ws.app.manager_gen_15a42559-f979-4ee6-b488-9fa1fb212c96/.cache/Echo.war!/WEB-INF/wsdl/Echo.wsdl

To suppress these information messages, we can change the logging configuration in the server.xml file as follows:

<logging traceSpecification="org.apache.cxf.*=warning=enabled"/>

The SOAPFaultException: The given SOAPAction test does not match an operation occurred when a client application sent a SOAP Action.

Note: test is the value of the soapAction attribute in the request HTTP header. Each operation in a SOAP web service can be associated with a SOAP Action String, such as in the WSDL binding or via an annotation. The web service client sends the SOAP Action string as a header with the request to match the operation of the web service provider. The error message is displayed in either of the following scenarios:

When the SOAP action that is sent by the client does not match the SOAP action of the operation.
When using WebSphere Application Server traditional as the JAX-WS client and Liberty as the JAX-WS service and the soapAction defined in the WSDL file equals empty value "".

To resolve this problem:

Make sure specified the same SOAP action for both the web service client and service provider.
Provide a valid value to the soapAction attribute defined in the WSDL file or do not use soapAction in the WSDL file.

The javax.xml.ws.WebServiceException occurred when we use the Service.create() method to create service.

This error occurs when we use the Service.create() method to create a service while the WSDL document is not provided. To use the Service.create() to create a service and getPort() method to get the port number, use the addPort() method to provide the binding information.

A sample example code is provided as follows on how to use the addPort() method:

QName serviceName = new QName("http://test.com/wssvt/acme/InsBusiness/", "MTOM11Service");
QName portName = new QName("http://test.com/wssvt/acme/InsBusiness/", "MTOM11Port");

// Setup the necessary JAX-WS artifacts
Service svc = Service.create(serviceName);
// Add the port in the service instance
svc.addPort(portName, SOAPBinding.SOAP11HTTP_MTOM_BINDING, mtom11URL); 

port = svc.getPort(portName, MTOMInterface.class);

The Null Response returned when the name attribute in @WebResult annotation is different from the name element in the WSDL document.

This problem occurs when we use a SEI class to define the name attribute for @WebResult annotation, and the SEI class supplies the WSDL location as follows:

@WebService(wsdlLocation="WEB-INF/wsdl/WebServiceIfc.wsdl")
public interface WebServiceRuntimeIfc { @WebMethod
        @WebResult(name="nononoreturn") 
        public String echo (String parm);
}

But the XML element declaration in the supplied WSDL document is as follows:

<xs:element name="echoResponse">
	<xs:complexType>
		<xs:sequence>
			<xs:element name="return" type="xs:string" minOccurs="0"/>
		</xs:sequence>
	</xs:complexType>
</xs:element>

Then the web service client gets a NULL response when the echo() method is called.

To resolve this problem, ensure that the name attribute of the @WebResult annotation matches the value of the name element in the WSDL document.

The JAX-WS client application failed to get the WSDL document changes from the server side.

The problem occurs when the web service client and service provider are in two different applications on Liberty, and the client needs to retrieve the WSDL document from the service provider dynamically. Because Liberty caches the WSDL definition when the WSDL document is accessed for the first time, which is different from the behavior in WebSphere Application Server traditional. By caching the WSDL definition in Liberty, applications can avoid connecting and parsing the WSDL document frequently.

To solve this problem, we must restart the client application so that the updated WSDL definitions can be reloaded.

JAXB warning during WSDL import

The following warning message can be encountered while you import a WSDL file.

[WARNING] unknown extensibility element or attribute "wsdl" (in namespace "http://www.w3.org/2000/xmlns/" (http://www.w3.org/2000/xmlns/%27) )

Troubleshooting WS-Security

This section describes some common solutions used to solve WS-Security problems.

Check the server.xml file to make sure that the required JAX-WS (jaxws-2.2) and security (appSecurity-2.0) features are configured correctly.
To protect your web service application with WS-Security, your JAX-WS application must contain a WSDL file that has an embedded WS-Security policy. There must be a PolicyReference to the embedded WS-Security policy in either the wsdl:binding or wsdl:operation sections or both.
If we use a callback handler for retrieving passwords to generate UsernameTokens or for retrieving passwords for private keys from keystore files, the password callback handler must be packaged and deployed as a user feature in Liberty.

Enable WS-Security trace

If the cause of the problem cannot be determined by the information in the error message, we can use the Liberty trace and logging facility to collect trace for the WS-Security component.

See Liberty: Trace and logging. We can use the following trace string to collect trace to debug WS-Security failures:

org.apache.ws.security.*=all:
org.apache.cxf.ws.security.*=all:
org.apache.cxf.ws.policy.*=all:
org.apache.xml.security.*=all:
com.ibm.ws.wssecurity.*=all

This section described some common WS-Security problems and solutions we can choose.

org.apache.cxf.ws.policy.PolicyException: None of the policy alternatives can be satisfied.

This error usually occurs when the WS-Security feature wsSecurity-1.1 is not defined in the server.xml file. To avoid this error, we must define the wsSecurity-1.1 feature in the server.xml file as follows:

<featureManager>
  <feature>usr:wsseccbh-1.0</feature>
  <feature>servlet-3.0</feature>
  <feature>appSecurity-2.0</feature>
  <feature>jaxws-2.2</feature>
  <feature>wsSecurity-1.1</feature>
</featureManager>

org.apache.cxf.ws.policy.PolicyException: No callback handler and no password available.

This error occurs when the WS-Security run time cannot retrieve a password that is required to generate a UsernameToken or to access a private key from the keystore. To avoid this error, check the following configuration:

Ensure that the password callback handler feature wsseccbh-1.0 is defined as a user feature in the server.xml file:

<featureManager>
  <feature>usr:wsseccbh-1.0</feature>
  <feature>servlet-3.0</feature>
  <feature>appSecurity-2.0</feature>
  <feature>jaxws-2.2</feature>
  <feature>wsSecurity-1.1</feature>
</featureManager>

Ensure that the callback handler property ws-security.callback-handler is defined in the <wsSecurityClient> or <wsSecurityProvider> elements of the server.xml file:

ws-security.callback-handler="<full class name of the callback handler>"

example:
ws-security.callback-handler="com.ibm.ws.wssecurity.example.cbh.CommonPasswordCallback"

Ensure that the password callback handler returns the correct password for the user name or the key alias specified in the server.xml file. The password must be clear text and not encoded or encrypted.

org.apache.ws.security.WSSecurityException: The signature does not cover the required elements (soap:Body).

This error occurs when a web service provider application expects the SOAP body in the request message to be signed, but the received SOAP message does not have the SOAP body signed. To avoid this error, ensure that your web service client is configured with the correct WS-Security policy that matches the web service provider's policy.

org.apache.ws.security.WSSecurityException: The signature or decryption was invalid.

This error occurs when the WS-Security run time fails to validate the signature or decrypt an encrypted message part in the received SOAP message. To avoid this error, verify that the correct keys are used when you configure WS-Security in the <wsSecurityClient> and <wsSecurityProvider> elements of the server.xml file. If a web service client uses Bob's public key to encrypt a message part, then the web service provider must have access to Bob's private key to decrypt the message. Similarly, if a web service client signs a message part using Alice's private key, then the web service provider must use Alice's public key to validate the signature.

org.apache.cxf.ws.policy.PolicyException: These policy alternatives cannot be satisfied.

This error occurs when the WS-Security run time cannot process the incoming SOAP message successfully using the WS-Security policy configured to process this request. To avoid this error, look at the messages that follow this exception to determine the specific WS-Security policy assertions that are causing this exception. For example, the log file might contain the following messages:

Caused by: org.apache.cxf.ws.policy.PolicyException: These policy alternatives can not be satisfied: 
{http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702}AsymmetricBinding: The encryption algorithm does not match the requirement
{http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702}InitiatorEncryptionToken
{http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702}RecipientEncryptionToken
at org.apache.cxf.ws.policy.AssertionInfoMap.checkEffectivePolicy(AssertionInfoMap.java:167)
at org.apache.cxf.ws.policy.PolicyVerificationInInterceptor.handle(PolicyVerificationInInterceptor.java:101)

In this case, the error occurred because the encryption algorithm used by the web service client does not match the encryption algorithm specified by the AlgorithmSuite assertion. The algorithm suites specified in the WS-Security policy of both the web service client and the web service provider must specify the same encryption algorithm.

javax.xml.ws.soap.SOAPFaultException: The message has expired (WSSecurityEngine: Invalid timestamp The security semantics of the message have expired).

This message occurs when the message timestamp is expired or the message was created with a timestamp in the future.

Apply fix packs and interim fixes to an archive install

If we installed our Liberty runtime environment from an archive file, rather than using Installation Manager, we must take special measures when you apply service updates.

See Apply a fix pack to a Liberty Java archive installation and Apply an interim fix to a Liberty archive installation.

Troubleshooting performance

This section describes some common performance problems and solutions we can choose.

High CPU usage by the application monitor.

This error can occur if the application monitor has many files under the apps/ directory and is polling too frequently.

To avoid this problem there are a number of things we can change.

Increase the value of the pollingRate attribute in the applicationMonitor element.
Update the server.xml to include an applicationMonitor element with an updateTrigger that is not polled.
Reduce the number of files under the apps/ directory.

For more information about the applicationMonitor element, see Control dynamic updates.

Troubleshooting collectives

This section describes a problem with collectives and the solution we must apply.

java.lang.IllegalArgumentException: CWWKX0219E: The MBean 'WebSphere:feature=collectiveController,type=CollectiveRepository,name=CollectiveRepository' does not have an operation by the name 'retrieveMemberRootCertificate'

A server with feature collectiveMember-1.0, a member, cannot register with a server with feature collectiveController-1.0, a controller, that is at an earlier level of Liberty. For example, a member at this beta level of Liberty cannot register itself with a controller at V8.5.5.2 Liberty.

With default logging, this problem is reported as a First Failure Data Capture (FFDC) in the FFDC logs of the member.

To fix the problem, we must move the controller to the same or higher level of Liberty as the member.

Troubleshooting SAML

This section describes a problem with SAML and the solution we must apply.

java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0: This exception can occur when you attempt multiple logins through an unsolicited Service Provider (SP) initiated request without removing the Identity Provider token (IdP).; To avoid this, add <httpSession invalidateOnUnauthorizedSessionRequestException="true" /> in the relevant unsolicited server.xml file.

java.lang.IllegalStateException: CWWKS0010E: While getting the caller principal, the caller subject was found to have more than one principal of type WSPrincipal. Only one WSPrincipal can exist in the subject. The names of the WSPrincipals are: {0}: This exception can occur if a SAML user has previously logged directly into an on-premises user registry. To avoid this problem, a SAML user must not directly login to an on-premises user registry.

Troubleshooting REST API Discovery

If the IBM REST API Discovery Explorer Try it out selection is invoked remotely and fails with a response code equal to 0, ensure that the fully qualified hostname or the IP address is returned to the IBM REST API Discovery Explorer in the Curl and Request URL associated with the GET, PUT, POST, or DELETE operation. If the fully qualified hostname or IP address is not in the Curl and Request URL, complete the following actions:

Add the variable element in server.xml and specify the fully qualified hostname. Here is the example of adding the variable element that specifies the fully qualified hostname in the server.xml file:
Specify the full computer name for the operating system as the fully qualified host name.
Verify that the fully qualified hostname is returned in the Curl and Request URL associated with the GET, PUT, POST, or DELETE operations in the IBM REST API Discovery Explorer.
See Set the default host name of a Liberty server and read about configuring your network in the IBM InfoSphere Information Server, Version 11.3.1 product documentation.

Troubleshooting applications not starting in an embedded Liberty server

Ensure that the Java process that starts the embedded Liberty server was started with the -javaagent JVM argument that pointed to the libertyInstallDir/bin/tools/ws-javaagent.jar. If the -javaagent JVM argument is not used the server runtime starts, but applications fail to start with no obvious exceptions.