Runtime migration troubleshooting

Review this page for troubleshooting tips if you encounter problems while migrating from an earlier version of WebSphere Process Server or IBM BPM.

The following sections describe specific errors and exceptions that might occur in a runtime version migration of WebSphere Process Server or IBM BPM and provide steps you can follow to understand and resolve these problems.

Errors that result from migration command failures are logged in the snapshot directory in files whose names begin with commandname and end with .error. Use these files to determine the exact place the problem occurred.


Application installation error

If you select the option for the migration process to install the enterprise applications that exist in the earlier version into the new IBM BPM V8.0.1 version, you might encounter some error messages during the application-installation phase of migration.

The applications that exist in the WebSphere Process Server or IBM BPM earlier version might have incorrect deployment information—usually, incorrect XML documents that were not validated sufficiently in previous WebSphere Process Server or IBM BPM runtime environments. The runtime environment now has an improved application-installation validation process and will fail to install these malformed EAR files. This results in a failure during the application-installation phase of BPMMigrateProfile and produces an "E:" error message.

If the application installation fails in this way during migration, you can do one of the following:


Application server error

After you migrate a managed node to V8.0.1, the application server might not start.

When you try to start the application server, you might see errors similar to those in the following example: 

[5/11/06 15:41:23:190 CDT] 0000000a SystemErr  R  
    com.ibm.ws.exception.RuntimeError:
com.ibm.ws.exception.RuntimeError:  org.omg.CORBA.INTERNAL: 
   CREATE_LISTENER_FAILED_4
vmcid: 0x49421000  minor code: 56  completed: No
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:198)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:139)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:460)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
[5/11/06 15:41:23:197 CDT] 0000000a SystemErr  R  at  
sun.reflect.DelegatingMethodAccessorImpl.invoke
    (DelegatingMethodAccessorImpl.java:43)

Change the port number at which the managed node's server is listening. If the dmgr is listening at port 9101 for ORB_LISTENER_ADDRESS, for example, the server of the managed node should not be listening at port 9101 for its ORB_LISTENER_ADDRESS. To resolve the problem in this example...

  1. On the administrative console, click Servers > Server Types > WebSphere application servers > server_name > Ports > ORB_LISTENER_ADDRESS.
  2. Change the ORB_LISTENER_ADDRESS port number to one that is not used.


    BPMSnapshotSourceProfile and BPMMigrateProfile errors

    If you encounter a problem when you are migrating from an earlier version of IBM BPM to V8.0.1, check your log files and other available information.

    • If the migration job fails before the BPMMigrateProfile step, rerun the migration job.

    • If the migration job fails in the BPMMigrateProfile step, the configuration of the new V8.0.1 server is partially updated, so re-create (or restore from backup) the new V8.0.1 server before rerunning the migration job.
    • Problems occur with a managed (federated) node migration.

      A federated node is the most complex node to migrate because it is essentially two migrations rolled into one. A federated node requires a migration of the node configuration information contained in the dmgr's master repository as well as the configuration information contained in the federated node. Federated node migration requires an active connection to the dmgr. If you have security enabled, it is essential that you follow the instructions that were generated when you created the migration jobs. The migration job must be submitted with a WebSphere Administrator user ID that has been properly configured for obtaining secure connections.

      Version 6.x node agents might display as not synchronized or not available when you change the dmgr node name in a mixed cell during migration to the V8.0.1 dmgr. Version 6.x node agents maintain a link to the version 6.x dmgr until they are restarted; therefore, they might fail to synchronize with the new dmgr. The discovery problem, which prevents automatic synchronization, occurs because the node agent is not yet aware of the dmgr name change that occurred during the migration. If you experience this problem, perform these steps on the node:

      1. Stop the node.

      2. Run the syncNode command.
      3. Restart the node.
    • Job fails during the application-installation phase of migration.

      If you select the option for the migration process to install the enterprise applications that exist in the earlier configuration into the new V8.0.1 configuration, you might encounter error messages during the application-installation phase of migration.

      The applications that exist in the earlier version might have incorrect deployment information—usually, incorrect XML documents that were not validated sufficiently in previous IBM BPM runtime environments. The runtime now has an improved application-installation validation process and will fail to install these malformed EAR files. This results in a failure during the application-installation phase of BPMMigrateProfile and produces an "E:" error message.

      If the application installation fails in this way during migration, you can do one of the following:

      • Fix the problems in the earlier applications, and then remigrate.
      • Proceed with the migration and ignore these errors.
        1. Restart the migration job in the FINISHUP step to allow the remaining migration functions to be performed.

          Do this by adding the RESTART=FINISHUP parameter to the job card and resubmitting the job.

        2. Later, fix the problems in the applications and then manually install them in the new V8.0.1 configuration using the administrative console or an installation script.

    • Out of space errors occur.

      The migration logs are located in temporary_directory_location/ nnnnn, where temporary_directory_location is the value that you specified when you created the migration jobs (where the default is /tmp/migrate) and nnnnn is a unique number that was generated during the creation of your migration jobs. Normally, the space requirements for the migration logs are small. If you enable tracing, however, the log files can be quite large. Enable tracing only after problems have been found. If tracing is required, try to only enable tracing related to the step in the process that is being debugged. This will help to reduce the space requirements.

      Tracing can be enabled when you create the migration jobs or by changing the variables in the migration JCL from disabled to enabled: 

      TraceState=enabled
profileTrace=disabled
preUpGradeTrace=disabled
postUpGradeTrace=enabled

      During migration, a backup copy of your earlier configuration is made. This backup becomes the source of the information being migrated. The default backup location is /tmp/migrate/ nnnnn. This location can be changed when you create the migration jobs. Depending on the size of the node being migrated, this backup can be quite large.

      If your temporary space is inadequate, you will need to relocate this backup.

    • Batch job time is exceeded.

      Each z/OS installation is different with respect to job classes and time limitations. Make sure you have specified appropriate job classes and timeout values on your job card.

    • Failures occur during server startup after migration.

      Review the instructions that were generated when you created the migration jobs. Verify that the JCL procedures have been copied over correctly to your PROCLIB, the RACF definitions have been created, the V8.0.1 libraries have been authorized, and, if required, your STEPLIB statements to the V8.0.1 libraries have been specified. Verify that the daemon process associated with your cell is at the appropriate level. The daemon process must be at the highest IBM BPM for z/OS version level of all servers that it manages within the cell.

      After migrating to a V8.0.1 cell that contains or interoperates with nodes that are not at version 6.0.1.3 or later, the cluster function might fail. When starting these earlier version application servers, you might see the following problems:

      • You might see a first failure data capture (FFDC) log that shows a ClassNotFoundException error message. This exception is thrown from the RuleEtiquette.runRules method and looks something like the following example: 
        Exception = java.lang.ClassNotFoundException
Source = com.ibm.ws.cluster.selection.SelectionAdvisor.<init>
probeid = 133
Stack Dump = java.lang.ClassNotFoundException: rule.local.server
at java.net.URLClassLoader.findClass(URLClassLoader.java(Compiled Code))
at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:106)
at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
at java.lang.Class.forName1(Native Method)
at java.lang.Class.forName(Class.java(Compiled Code))
at com.ibm.ws.cluster.selection.rule.RuleEtiquette.runRules
(RuleEtiquette.java:154)
at com.ibm.ws.cluster.selection.SelectionAdvisor.handleNotification
(SelectionAdvisor.java:153)
at com.ibm.websphere.cluster.topography.DescriptionFactory$Notifier.run
(DescriptionFactory.java:257)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1462)

      • You might see a java.io.IOException that looks something like the following example: 
        Exception = java.io.IOException
Source = com.ibm.ws.cluster.topography.DescriptionManagerA. update probeid
 = 362
Stack Dump = java.io.IOException
at com.ibm.ws.cluster.topography.ClusterDescriptionImpl.importFromStream
(ClusterDescriptionImpl.java:916)
at com.ibm.ws.cluster.topography.DescriptionManagerA.update
(DescriptionManagerA.java:360)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java(Compiled Code))
at java.io.DataInputStream.readUTF(DataInputStream.java(Compiled Code))
at com.ibm.ws.cluster.topography.KeyRepositoryImpl.importFromStream
(KeyRepositoryImpl.java:193)

      During migration, V8.0.1 cluster information is distributed throughout the cell. Earlier nodes that are not at version 6.0.1.3 or later fail to read this information. To avoid this problem, upgrade all earlier nodes that will be contained in or interoperating with an earlier cell to version 6.0.1.3 or later before migrating your dmgrs to V8.0.1.

    After migration, carefully review the job output and log files for errors.

    IBM BPM provides an interactive problem control system (IPCS) verb exit to help you to format information from dumps of IBM BPM processes. This verb exit was named CBDATA, which was an alias of the real module name, in earlier versions. In V8.0, that alias was removed. In V8.0 and later, therefore, use the real name of this verb exit, BBORDATA, instead of the alias.

    If you migrate a node to V8.0.1 and then discover that you need to revert back to an earlier version, see Rolling back your environment.

    If none of these steps solves the problem, see "Troubleshooting and support for IBM BPM" for additional troubleshooting resources, including information about how to contact IBM Support.


    Business Space migration exception

    By default, the IBM BPM server Business Space component attempts to read every user attribute from the user Lightweight Directory Access Protocol (LDAP) repository; however, the WebSphere Application Server WAP Identity Module (WIM) component has trouble resolving some of the attribute fields that point to a removed entity in the user repository.

    If you see errors in the profile_root/logs/server_name/systemout.log file after migration similar to com.ibm.websphere.wim.exception.WIMSystemException: CWWIM1013E The value of the property secretary is not valid for entity, follow steps 1 and 2 to limit the number of user attributes that are required for IBM BPM to run successfully.

    1. In the administrative console, navigate to: Resources > Resource Environment > Resource environment providers > Mashups_ConfigService > Custom properties.
    2. Modify or add the following property and set the value to LIMITED:

      Name - com.ibm.mashups.user.userProfile

      Value - LIMITED


    Communication with dmgr error

    Sometimes the migration process can fail because of insufficient resources on the machine. If the migration fails, check the log file to see if the following message appears: 

    "MIGR0494E: An unexpected error occurred during communication with the Deployment Manager, 
the migration cannot continue. Resolve the error and rerun the WASPreUpgrade tool to 
create a new backup directory."

    If you see this message in the log file, check the disk space on the machine and the memory and CPU utilization. If possible, stop some other processes on the machine to free up machine resources and rerun the migration command that has failed.


    ConnectorException

    When migrating a managed node, if you see a ConnectorException as follows, ensure that your dmgr is running and rerun the command. 

    MIGR0380E: The JMX connection is not established with the dmgr node qaxs06, 
using connector type of SOAP on port 8879. The WASPostMigration program is now closing. No 
changes are made to the local Application Server environment.
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot 
create a SOAP connector to connect to host qaxs06 at port 8879.
com.ibm.ws.migration.utility.UpgradeException: 
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot 
create a SOAP connector to connect to host qaxs06 at port 8879.


    Exceptions: database connectivity, loading, or missing class

    Never change any WebSphere Application Server variables that are configured as part of profile creation.

    If you modify these values incorrectly in the old profile, you might get database connectivity, loading, or other missing class exceptions, such as:

    10/25/08 13:22:39:650 GMT+08:00] 0000002e J2CUtilityCla E J2CA0036E: An exception occurred while invoking method setDataSourceProperties on com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl used by resource jdbc/com.ibm.ws.sib/ewps6101.Messaging-BPC.cwfpcCell01.Bus : com.ibm.ws.exception.WsException: DSRA0023E: The DataSource implementation class "com.ibm.db2.jcc.DB2XADataSource" could not be found.DB2,

    SQL Embedded JDBC drivers are bundled with the IBM BPM product installation. If you need to change these drivers to any higher version, you must copy drivers on the same location where they exist in the product installation, as follows:

    • DB2:% was.install.root%/jdbcdrivers/DB2
    • SQL:% was.install.root%jdbcdrivers/SQLServer
    • Oracle:% was.install.root%jdbcdrivers/Oracle

    If you need a new JDBC provider and datasource for your application, you can create these resources by selecting a valid jdbcclasspath and setting the WebSphere Application Server variable accordingly.

    For example, if you need DB2 at a cell level that does not exist earlier in your installation, you could use the following procedure:

    1. In the administrative console, navigate to: Resources > JDBC > JDBC Providers > DB2 Universal JDBC Driver Provider (XA).

    2. In the Class path box, set the following paths:

      • DB2UNIVERSAL_JDBC_DRIVER_PATH =%was.install.root%/universalDriver_wbi/lib
      • DB2UNIVERSAL_JDBC_DRIVER_NATIVEPATH=""

      If you need your own drivers, set the following path: DB2UNIVERSAL_JDBC_DRIVER_PATH=%myDriverLocation%


    LifeCycleServiceError

    The LifeCycleService error can occur as a postmigration exception if changesets in the database repository are in the active state.

    This error applies only to a version 6.2 to 7.x migration scenario.

    You can avoid this error by checking for changes before migrating WebSphere Process Server version 6.2 to 7.x, as described in Runtime premigration checklist.


    Missing step BPMMigrateCluster

    If the ND migration process misses the step BPMMigrateCluster for a cluster, the servers in that cluster show the following error message in the server log file: 

    [3/4/10 10:52:20:767 CST] 00000000 WsServerImpl E WSVR0009E: Error occurred during startup
com.ibm.ws.exception.RuntimeError: CWMCO0908E: The cluster <cluster name> has not been migrated to version that's compatible with node suse40Node01. Please execute BPMMigrateCluster command on the dmgr profile to complete cluster migration.
at com.ibm.bpm.migration.cluster.detection.DetectClusterMigration.start(DetectClusterMigration.java:135)
at com.ibm.ws.runtime.component.ContainerHelper.startComponents(ContainerHelper.java:538)
at com.ibm.ws.runtime.component.ContainerImpl.startComponents(ContainerImpl.java:627)

    Perform the following actions to correct this error:

    1. Run the BPMMigrateCluster command on the dmgr profile to migrate the cluster configuration.

    2. Perform a syncNode operation for all managed nodes that participate in the cluster.
    3. Restart the cluster.


    Out of memory error

    If either the BPMSnapshotSourceProfile or BPMMigrateProfile command-line utility fails because of out of memory problems, you can increase the heap size to a number that takes into consideration the size and scope of the environment being migrated, as well as what the machine will allow.

    For instructions on how to increase the heap size, use the procedure described in Solution 4 of the following technote: Handling certain Out of Memory conditions when migrating an earlier version of WebSphere Application Server to V6.0.2, V6.1, or 7.0.


    Custom node migration error

    When you use the migration wizard to migrate a custom node from an earlier version of IBM BPM to IBM BPMV8.0.1, you might receive the following certificate exchange message in the custom nodes WASPostUpgrade.Custom01.<timestamp>.log: 

    MIGR0404W: Do not use the node agent in the old configuration. It has been disabled.
CWPKI0310E: The <localKeyStoreName> specified as "ClientDefaultTrustStore" was
           not found on the client.
CWPKI0314E: The following error is returned from an exception: Exception thrown
           in RequiredModelMBean while trying to invoke operation            retrieveSigners
CWPKI0304E: The <remoteKeyStoreName> specified as            "NodeDefaultTrustStore" was not found on the server.
[03/26/2012 20:08:07:942 EDT] MIGR0519I: The certificate exchange related messages can be ignored.
    The certificate exchange messages can be ignored.


    Servlet error

    In a ND environment, if the error SRVE0026E: [Servlet Error]-[com/ibm/wbiservers/brules/BusinessRuleManager]: java.lang.NoClassDefFoundError occurs when you access the Business Process Rules Manager after migrating, you must manually install the Business Process Rules Manager application on the deployment target before continuing with normal migration of that node. See the Business Process Rules Manager section in the What gets migrated topic for more information.


    Unable to pause the Event Manager

    During migration from WebSphere Lombardi Edition 7.2 or IBM BPM 7.5 or 7.5.1, you might see the following error message: 

    Unable to pause the Event Manager
    Check to see if the messaging engine at the Performance Data Warehouse Bus is started normally. If it is not, configure and start it normally, and try the migration again.


    Synchronization error

    If synchronization fails when you migrate a managed node to IBM BPM V8.0.1, the server might not start.

    You might receive messages similar to the following when you migrate a managed node to V8.0.1: 

    ADMU0016I: Synchronizing configuration between node and cell.
ADMU0111E: Program exiting with error:
           com.ibm.websphere.management.exception.AdminException: ADMU0005E:
           Error synchronizing repositories ADMU0211I: Error details may be seen in the file:
           /opt/WebSphere/62AppServer/profiles/AppSrv02/logs/syncNode.log
MIGR0350W: Synchronization with the dmgr using the SOAP protocol 
 failed.
MIGR0307I: The restoration of the previous WebSphere Application Server 
 environment is complete.
MIGR0271W: Migration completed successfully, with one or more warnings.
    These messages indicate the following:

    • Your dmgr is at a V8.0.1 configuration level.

    • The managed node that you are trying to migrate is at a V8.0.1 configuration level on the dmgr's repository (including applications).

    • The managed node is not quite complete because you did not complete the syncNode operation.

    Perform the following actions to resolve this issue:

    1. Rerun the syncNode command on the node to synchronize it with the dmgr.

      See the syncNode command .

    2. Run the GenPluginCfg command.

      See the GenPluginCfg command .


    Warning: There are date after end of archive

    If the BPMCreateDatabaseUpgradeUtilities command is run on a UNIX platform but the generated compressed file would be used on a Windows platform for execution of the BPMGenerateUpgradeSchemaScripts command, during extraction of the compressed file you might see the following warning: Warning: There are date after end of archive. This warning is harmless, and the extracted content and command should work successfully.


    WebSphere Process Server or IBM BPM client migrations

    When migrating WebSphere Process Server or IBM BPM client profiles from an earlier version to a full server IBM BPM V8.0.1 installation, the target profile augmentation is not correct. Applications on the target profile might not work correctly. To correct the problem, use manageprofiles.sh to add the augmentation for the target_INSTALL_ROOT/profileTemplates/SCA/*.sdo template, where the "*" symbol represents "default" for standalone and "managed" for federated profiles.


    WSDL validation exception

    If the BPMMigrateProfile command fails with the following WSDL validation exception, it means that a WSDL file in the application that failed to install has an input element declaration that is not defined within an operation. To fix this problem, you must either define the input element declaration or remove it from the WSDL file.

    WSDL validation exception 

    java.io.IOException: javax.wsdl.WSDLException: WSDLException (at /wsdl:definitions
/wsdl:import/wsdl:definitions/wsdl:input): faultCode=INVALID_WSDL: Encountered 
illegal extension element '{http://schemas.xmlsoap.org/wsdl/}input' in the context 
of a 'javax.wsdl.Definition'. Extension elements must be in a namespace other than 
WSDL's.
javax.wsdl.WSDLException: WSDLException (at /wsdl:definitions/wsdl:import/wsdl:
definitions/wsdl:input): faultCode=INVALID_WSDL: Encountered illegal extension 
element '{http://schemas.xmlsoap.org/wsdl/}input' in the context of a 'javax.wsdl.
Definition'. Extension elements must be in a namespace other than WSDL's.

    How to fix the problem

    Use the following procedure to fix the problem:

    1. Locate the WSDL file in the application that failed to install. The WSDL file that is failing in validation has an input element declaration that is not defined within an operation.

      Sample of a failed WSDL file:

      The declaration for getLastSellPriceRequest is not defined under the wsdl:operation declaration. 

      wsdl:portType name="EnrollIntf"
wsdl:operation name="Enrollment"
wsdl:input message="tns:EnrollmentRequestMsg" name="EnrollmentRequest"/
wsdl:output message="tns:EnrollmentResponseMsg" name="EnrollmentResponse"/
/wsdl:operation
/wsdl:portType
      wsdl:input name="getLastSellPriceRequest"
wsdlsoap:header message="tns:EnrollmentRequest" part="soap_header" use="literal"/
wsdlsoap:body parts="EnrollReq" use="literal"/
/wsdl:input
    2. Make the appropriate change to the input declaration, depending on whether the input declaration file is needed or not.

      • If the input declaration is needed, move it under the operation that uses it.

      • If the input declaration is not needed, remove it from the WSDL file.

    3. Update the application in source environment.
    4. Verify that the application works in the source environment.

    5. Perform the migration steps again, starting with the BPMSnapshotSourceProfile command or the IBM BPM profile migration wizard.
    Migrating your IBM BPM Advanced V7.5.x or WebSphere Process Server V7.x or V6.2.x runtime
    ol> br> ul> html>