WAS v8.5 > Reference > Troubleshooting tips

WS-Notification troubleshooting tips

Tips for troubleshooting WS-Notification publish and subscribe messaging for web services.

To enable trace for WS-Notification, set the application server trace string to SIBWsn=all=enabled:com.ibm.ws.sib.webservices.*=all=enabled. If we encounter a problem that you think might be related to WS-Notification, we can check for error messages in the WebSphere Application Server dmgr console, and in the application server SystemOut.log file. We can also enable the application server debug trace to provide a detailed exception dump.

IBM recommends using the HPEL log and trace infrastructure. With HPEL, one views logs using the LogViewer command-line tool in PROFILE/bin.

A list of the main known restrictions that apply when using WS-Notification is provided in WS-Notification: Known restrictions.

WAS system messages are logged from a variety of sources, including application server components and applications. Messages logged by application server components and associated IBM products start with a unique message identifier that indicates the component or application that issued the message. The prefix for the WS-Notification component is CWSJN.

The Troubleshooter reference: Messages topic contains information about all WAS messages, indexed by message prefix. For each message there is an explanation of the problem, and details of any action that we can take to resolve the problem.

Here is a set of tips to help troubleshoot commonly-experienced problems:


A JAX-WS application that is a raw consumer of brokered notifications must recognize a notification broker SOAP action

JAX-WS supports action-based dispatch, and JAX-WS raw consumer applications must accept the http://docs.oasis-open.org/wsn/bw-2/NotificationConsumer/Notify action URI. We can enable this in any of the following ways:


The PublisherRegistrationManager.wsdl file is not successfully parsed by wsimport unless you include a JAX-WS bindings file

When you publish the WSDL files for a WS-Notification application to a compressed file, then run the wsimport command against the PublisherRegistrationManager.wsdl file, the following fault message is displayed:

[ERROR] the following naming conflicts occurred: 
com.ibm.websphere.wsn.publisher_registration_manager.ResourceNotDestroyedFault
line 2 of file:/path_to_wsdl/PublisherRegistrationManager.wsdl

This fault occurs because the WSDL for the publisher registration manager uses both the WS-Notification and the WS-ResourceLifetime specifications; both of which refer to a ResourceNotDestroyedFault element that shares the same message name. Here is the relevant part of the publisher registration manager WSDL file:

  <wsdl:operation name="DestroyRegistration">     <wsdl:input name="DestroyRegistrationRequest" message="wsn-brw:DestroyRegistrationRequest" 
      wsam:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationRequest" 
      wsaw:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationRequest"/>     <wsdl:output name="DestroyRegistrationResponse" message="wsn-brw:DestroyRegistrationResponse" 
      wsam:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationResponse" 
      wsaw:Action="http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationResponse"/>     <wsdl:fault name="ResourceUnknownFault" message="wsrf-rw:ResourceUnknownFault" 
      wsam:Action="http://docs.oasis-open.org/wsrf/fault" 
      wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/>     <wsdl:fault name="ResourceNotDestroyedFault" message="wsn-brw:ResourceNotDestroyedFault" 
      wsam:Action="http://docs.oasis-open.org/wsn/fault" wsaw:Action="http://docs.oasis-open.org/wsn/fault"/>   </wsdl:operation> 
<!-- Some parts have been omitted --> 
<!-- An extract from WS-ResourceLifetime -->   <wsdl:operation name="Destroy">     <wsdl:input name="DestroyRequest" message="wsrf-rlw:DestroyRequest" 
      wsam:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyRequest" 
      wsaw:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyRequest"/>     <wsdl:output name="DestroyResponse" message="wsrf-rlw:DestroyResponse" 
      wsam:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyResponse" 
      wsaw:Action="http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyResponse"/>     <wsdl:fault name="ResourceNotDestroyedFault" message="wsrf-rlw:ResourceNotDestroyedFault" 
      wsam:Action="http://docs.oasis-open.org/wsrf/fault" 
      wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/>     <wsdl:fault name="ResourceUnknownFault" message="wsrf-rw:ResourceUnknownFault" 
      wsam:Action="http://docs.oasis-open.org/wsrf/fault" 
      wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/>     <wsdl:fault name="ResourceUnavailableFault" message="wsrf-rw:ResourceUnavailableFault" 
      wsam:Action="http://docs.oasis-open.org/wsrf/fault" 
      wsaw:Action="http://docs.oasis-open.org/wsrf/fault"/>   </wsdl:operation>

To resolve this naming conflict, you must include the JAX-WS bindings file ibm-wsn-jaxws.xml as an argument to the wsimport command. This bindings file ensures the conflicting elements are mapped to different class names.

The ibm-wsn-jaxws.xml file is located in the app_server_root/util directory. For example: c:\was\util\ibm-wsn-jaxws.xml. This bindings file expects to find the WSDL file to which it refers in the same directory as itself, so before running the wsimport command you must copy the bindings file to the directory that holds your PublisherRegistrationManager.wsdl file. Here is an example of how to run the wsimport command to include the ibm-wsn-jaxws.xml file:


WS-Notification service receives a triggerActionNotSupportedFault

You have a JAX-WS demand-based publisher registered with a v7.0 type of WS-Notification service. When the service invokes any of the operations on the PausableSubscriptionManager interface of the publisher, the publisher responds with a triggerActionNotSupportedFault exception message. The operations that trigger this message are Renew, Unsubscribe, PauseSubscription or ResumeSubscription.

You see messages similar to the following text in the SystemOut.log file for the server. In this example the fault message is triggered in response to the broker attempting to unsubscribe from the demand-based publisher.

triggerActionNotSupportedFault triggerActionNotSupportedFault: messageContext: 
[MessageContext: logID=urn:uuid:13616A3EB4F278A3DC1221827497002] problemAction: 
http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest

CWSJN1029W: An attempt was made to perform an operation on a remote NotificationProducer
 located with endpoint Address[[xxx/prod_service/NotificationProducerPort]], 
ReferenceParams[] but the operation could not be completed. This operation will be 
retried automatically in 2 seconds. 
The operation failed due to com.ibm.ws.sib.wsn.webservices.WSNWSException: 

CWSJN5035E: An internal error occurred: Unable to unsubscribe from remote publisher 
at endpoint reference Address[[xxx/prod_service/PausableSubscriptionManagerPort]], 
ReferenceParams[] due to javax.xml.ws.soap.SOAPFaultException: 
The [action] cannot be processed at the receiver.

The server continues to unsuccessfully attempt to unsubscribe from the publisher at ever-increasing time intervals.

When the WSDL file for the demand-based publisher does not specify a SOAP action pattern, WS-Addressing generates one by default. If wer publisher uses the PausableSubscriptionManager port type for its binding, the default action patterns generated by WS-Addressing do not match the actions defined by the WS-Notification specification.

This problem only occurs if your publisher uses the PausableSubscriptionManager port type. If wer publisher uses the SubscriptionManager port type, then the default action patterns generated by WS-Addressing match those in the WS-Notification specification.

To resolve this problem, in the WSDL file for the demand-based publisher explicitly specify the SOAP action to use for each of the operations on the PausableSubscriptionManager interface. The action URI to use for each operation is defined in the Web Services Base Notification 1.3 (WS-BaseNotification) specification available from http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsn For example, the WS-Addressing action for the Unsubscribe operation is defined in the specification as http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest, therefore use this action for the Unsubscribe operation in your publisher WSDL file. Here is an excerpt from the binding section of such a WSDL file:

<wsdl:binding name="PausableSubscriptionManagerBinding" type="wsn-bw:PausableSubscriptionManager"> <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" /> 
<wsdl:operation name="Unsubscribe"> <soap:operation soapAction="http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest" />

Similarly specify the corresponding actions for the Renew, PauseSubscription and ResumeSubscription operations in your publisher WSDL file.


There are "Connection timed out" errors in your broker server logs

If we see WebServicesFault( "Connection timed out" ) errors in your WS-Notification broker server logs, and we have a large number of consumers subscribed to your WS-Notification service, the broker might have exceeded the maximum number of connections in the HTTP outbound connector connection pool when sending outbound notification messages to subscribed consumers.

To increase the number of outbound HTTP connections in the pool, set the com.ibm.websphere.webservices.http.maxConnection custom property on the server or servers on which your broker is running. For more information about this property, see HTTP transport custom properties for web services applications.


There are "Out of memory" errors in your broker server logs

If we see "Out of memory" errors in your WS-Notification broker server logs, and we have a large number of consumers subscribed to your WS-Notification service, the broker might have exceeded the available JVM heap size allocated to the server on which it is running.

To increase the maximum heap size for the server or servers on which your broker is running, see Tune the IBM virtual machine for Java.


WAS v6.1 client applications must handle the additional error conditions

In WAS v6.1, support for WS-Notification is based on a pre-final approval public review draft of the WS-Notification standards. In later versions, this support is extended to cover the final approved standards. The differences between the WS-Notification public review drafts and final standards are as follows:

The WSDL and schema files shipped with WAS v7.0 or later are updated to reflect the final v1.3 WS-Notification standards.

You need not change your existing WS-Notification services. Your existing client applications will also continue to work unchanged, but if they are now also working with new WS-Notification services, and you want them to explicitly handle the new fault conditions, then regenerate the client stubs using the WSDL file from the new service.


Exception occurs because the SDO repository is not configured correctly

If we try to create a v6.1 WS-Notification service, and you get the following stack trace, then the SDO repository is not configured correctly. To resolve this problem, see Install and configure the SDO repository.

java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: 
Failed to store WSDL located at http://www.ibm.com/websphere/wsn/notification-broker 
due to the following exception: com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: 
CWSWS1007E: The following exception occurred: 
com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: 
javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; 
nested exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: 
javax.transaction.TransactionRolledbackException: ; nested exception is: 
javax.ejb.TransactionRolledbackLocalException: ; nested exception is: 
com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: 
PMGR1014E: Exception occurred when getting connection factory: 
com.ibm.websphere.naming.CannotInstantiateObjectException: 
threw NameNotFoundException while the JNDI NamingManager was processing a 
javax.naming.Reference object. [Root exception is javax.naming.NameNotFoundException: 
Context: smeagolNode03Cell/nodes/smeagolNode03/servers/server1, name: 
jdbc/com.ibm.ws.sdo.config/SdoRepository: 
First component in name com.ibm.ws.sdo.config/SdoRepository not found. 
[Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: 
IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.


Multiple messages are received by a notification consumer for each event notification

In some situations you might receive more notifications at a given notification consumer than the number of event notifications that have been inserted into the notification broker by a publisher. For example you might publish 4 messages, and receive 8, 12, 16 (or some other multiple of four) messages at the notification consumer.

This is usually caused by there being two or more active subscriptions that target the notification consumer - a situation that can occur if the subscriber application is run more than once. Each time the Subscribe operation is called, a new subscription must be created by the notification broker (see section 4.2 of the Web Services Base Notification specification), which causes duplicate messages to be delivered if a previous subscription exists.

To check whether this is what is happening, examine the SubscriptionReference property of the notifications received by the notification consumer. This endpoint reference contains the identifier of the subscription that caused the notification to be sent. If we find several different subscription identifiers, then there is more than one subscription active.

Subscriber applications should tidy up subscriptions when they are not required (or register them with a timeout), however we can tidy them up administratively using the runtime panels as described in List or deleting active WS-Notification subscriptions.


Problems can occur when deleting administered subscribers and messaging engines

Delete administered subscribers

You should be wary of deleting and recreating messaging engines on bus members for which WS-Notification-administered subscribers have been configured, because in some cases this can leave the remote web service subscription active (and passing notification messages to the local server) even though there is no longer any record of it.

To avoid this situation you should delete the WS-Notification configuration, or just the administered subscribers, in a separate step to deleting the messaging engine. When the dynamic configuration update is then processed, or the server restarted, the remote web service subscription is tidied up cleanly.

This problem does not occur if it is only the WS-Notification configuration that is modified; it only occurs if the messaging engine is also deleted.


There are differences when using bus destinations with v6.1 WS-Notification services

Usually, a bus destination is used as described in Bus destinations. However, this is not the case for v6.1 WS-Notification services. The destination associated with a v6.1 WS-Notification service does not relate to the topics for which the WS-Notification service can handle requests, and you should not alter or mediate the destination. In WS-Notification, the configuring of topics is handled through topic namespaces. For more information, see Create a new WS-Notification permanent topic namespace.

When creating a v6.1 WS-Notification service, the wizard configures three service integration bus inbound services for the WS-Notification service, one for each of the three WS-Notification service roles:

These inbound services are defined on the same service integration bus as the v6.1 WS-Notification service, and each of these inbound services refers to the same bus destination.


An inbound (application to broker) notification is not successful

Applications that want to publish event notifications into the broker make use of the Notify operation. This is defined as a one-way (web services) operation which means that it is not possible to return a fault (exception) if it is not possible to complete the operation. Thus the application will assume the notification was successful, but subscribing applications will not receive the notification message.

The notification might be unsuccessful because of an application error (invalid topic syntax), or a mismatch between the application code and the server configuration (using an undefined topic namespace). Specific reasons for which an inbound notification might not succeed include the following:

You should monitor this type of exception closely, because it might indicate a denial of service attack and certainly indicates the application is not functioning correctly. The first time an inbound notification fails from a particular producing application, a warning message is sent to the SystemOut.log of the server. If there are further notification failures for that producer, subsequent timed warning messages are logged at 30 minute intervals. Additional information is provided with each timed message to indicate how many failed notifications were received for that producer during the 30 minute interval.

When the system generates each warning message, it identifies the producing application through one of two identifiers:

The system cannot identify the host IP address in all cases. For example, for SOAP over JMS transports the IP address of the originating host is not available or applicable.


Outbound (broker to application) notification is not successful

An outbound web service invocation (broker to application) does not succeed when a remote application is unavailable for invocation. This might be the result of an application failure, a network error, or a firewall configuration issue. When event notifications are not passed to subscribed applications, messages build up on the subscriptions held on the server. The messages held on a given subscription can be observed using the runtime panels as described in List or deleting active WS-Notification subscriptions. Subscriptions for which the most recent event notification attempt has failed in this way are marked as being in ERROR state when viewed in the WS-Notification subscription runtime administration panel.

If the WS-Notification service point does not successfully notify a NotificationConsumer application, a warning message is sent to the application server SystemOut.log and the subscription is told to wait for 2 minutes. Reasons for a failure of this type might be the remote web service is not currently available, or that network conditions prevent contact between the local server and the service.

After 2 minutes, the notification is retried. If delivery is still not possible then the subscription is put back into a wait state for another 2 minutes. If the failure is caused by a transient I/O error, this pattern is repeated indefinitely, until the notification is either successfully delivered or you delete the subscription. If the error is caused by an application failure on the remote side then the notification will be retried up to the number of times defined in the "Maximum failed deliveries" setting of the service integration bus topic space destination from which the message is being received. After the first warning message is output to the SystemOut.log, subsequent timed warning messages are logged at 30 minute intervals.


Tidying up stateful resources that are not automatically deleted

The act of subscribing to the broker or registering a publisher creates a stateful resource on the server that consumes system resources while it is active. Usually an application specifies a termination time as part of the act of creating these resources, and thus they are automatically deleted when the termination time is reached. However it is also possible for the application to request an infinite lifetime for the resource. If this is done then it is possible for resources to remain on the server indefinitely even though the application might never be coming back to use (or destroy) the resource.

We can view the stateful resources (subscriptions and publisher registrations) using the runtime panels described in Interacting at run time with WS-Notification. These panels also provide the ability to administratively delete the items if required. Only do this if you are sure the application is no longer using the resource because it will cause application failures if the resource is referenced after being deleted.


Durable subscription that was created by WS-Notification cannot be deleted when using the service integration bus panel

When creating a subscription using a WS-Notification application, that is using the Subscribe operation, one or more durable subscriptions are created in the relevant service integration bus topic space destination. We can view these durable subscriptions in the service integration bus runtime panels for the publication point.

The runtime panels for the publication point also provide the ability to delete one or more durable subscriptions. However, if we use this function to delete a subscription that was created by a WS-Notification application, the delete operation does not succeed. This is because the WS-Notification implementation maintains an active consumer for this durable subscription for the duration of the time the server is running, and a durable subscription cannot be deleted if an active consumer is present.

This deletion restriction also applies to durable subscriptions created by other applications, such as JMS applications.

To delete a subscription that was created by a WS-Notification application, use the runtime panels provided by the WS-Notification implementation, as described in Interacting at run time with WS-Notification. This approach closes the active consumer and automatically deletes the related service integration bus durable subscriptions.


Administrative stop of a messaging engine

WAS depends on being able to access a running service integration bus messaging engine to send and receive messages, and to create and retrieve state for the various web service resources created.

We can stop a messaging engine using the MBean interface or runtime panels. This prevents WS-Notification from successfully servicing any requests from applications that might come in during the time the messaging engine is stopped. In this situation, error messages are logged as described in An inbound (application to broker) notification is not successful and Outbound (broker to application) notification is not successful. When you stop a messaging engine, all WS-Notification processing stops and all messaging applications cease to function. When you restart the messaging engine, WS-Notification processing resumes.


Failures as a result of changes in topic space and topic namespace configurations

The WS-Notification configuration artifacts often depend on objects defined in other areas of the server configuration. For example (for v6.1 WS-Notification services) the endpoint listeners through which application requests are received, and the service integration bus topic spaces to and from which messages are sent.

The following items describe the action that is taken by the WS-Notification runtime code when it meets relevant changes in the objects upon which it depends.

Delete a service integration bus topic space

The service integration bus topic space is the primary messaging object upon which WS-Notification depends at run time. Notification messages from an application are published to the topic space specified by the (permanent) topic namespace mapping specified by the administrator.

Deleting a service integration bus topic space has the following effects upon new and existing WS-Notification applications:

  • RegisterPublisher requests that use a WS-Notification topic namespace that references the deleted topic space receive a TopicNotSupportedFault error message.
  • Notify requests for a topic associated with the deleted topic space do not publish the message to the topic space (because it has been deleted). The application is not informed because no faults are thrown by the Notify operation (see An inbound (application to broker) notification is not successful).
  • Subscribe requests that use a WS-Notification topic namespace that references the deleted topic space receive a SubscribeCreateFailedFault error message.
  • No further messages are delivered to applications that have existing subscriptions to the deleted topic space. The existing subscription is deleted, and any attempt to invoke operations on the subscription (for example getCreationTime) results in a ResourceUnknownFault error message.
  • Deleting and recreating a service integration bus topic space is considered as two separate steps. Existing subscriptions are deleted in response to the first step, and therefore do not exist when the topic space is recreated.

Delete a permanent topic namespace mapping

Deleting the topic namespace mapping that was used to establish a (currently active) subscription has the same effect as deleting the underlying service integration bus topic space as defined previously, and subscriptions that were created using this namespace mapping are deleted.

Publisher registrations and pull points associated with the deleted topic namespace mapping are also deleted.

Change a permanent topic namespace mapping

The fields of a permanent topic namespace mapping are read-only fields, so the only way to "change" the fields is to delete the namespace mapping and recreate it with new values. The effect of deleting a permanent topic namespace mapping is described in the previous item.


Unable to create a v6.1 WS-Notification service without a configured SDO repository

When creating a v6.1 WS-Notification service, WSDL documents are saved into the SDO repository. You will see the following exception message if you try to create a v6.1 WS-Notification service using the dmgr console, or through scripting, before successfully configuring the SDO repository.

java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: Failed to 
store WSDL located at http://www.ibm.com/websphere/wsn/notification-broker due to the following 
exception:
    com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: CWSWS1007E: The following exception 
        occurred: com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: 
        javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; nested 
        exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: 
        javax.transaction.TransactionRolledbackException: ;
    nested exception is: javax.ejb.TransactionRolledbackLocalException: ;
    nested exception is: com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: PMGR1014E: 
        Exception occurred when getting connection factory: 
        com.ibm.websphere.naming.CannotInstantiateObjectException: threw NameNotFoundException while 
        the JNDI NamingManager was processing a javax.naming.Reference object.
    [Root exception is javax.naming.NameNotFoundException: Context: 
        KADGINNode01Cell/nodes/KADGINNode01/servers/server1, name: 
        jdbc/com.ibm.ws.sdo.config/SdoRepository: First component in name 
        com.ibm.ws.sdo.config/SdoRepository not found.
    [Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: 
        IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.
For details on how to configure the SDO repository, see Install and configure the SDO repository.


Exception occurs when a JAX-WS client that does not have internet access attempts to contact a WS-Notification service

The client application usually resolves the WSDL parts for the service by following web links. If the machine on which the client runs does not have internet access, an exception similar to the following example occurs:

WSDLException (at /definitions/import[1]): faultCode=OTHER_ERROR: 
Unable to resolve imported document at 'http://docs.oasis-open.org/wsn/brw-2.wsdl', 
relative to 'http://localhost:9082/WSNService1WSNServicePt1NB/Service/WEB-INF/wsdl
/NotificationBroker.wsdl': java.net.UnknownHostException: docs.oasis-open.org

Configure the client to use a local copy of the WSDL file instead, by following the instructions in the topic Configure a JAX-WS client to resolve a WS-Notification service WSDL without following web links.


Related concepts:

WS-Notification


Related


Troubleshoot applications with HPEL
Use WS-Notification for publish and subscribe messaging for web services
Secure WS-Notification


Reference:

WS-BaseNotification v1.3 OASIS Standard

XML Path Language (XPath) v1.0

XQuery 1.0 and XPath 2.0 Formal Semantics


+

Search Tips   |   Advanced Search