Troubleshooting

+

Search Tips   |   Advanced Search

  1. Enable account access for the IBM service representative
  2. Obtaining a collection set of problem determination information for IBM support
  3. Create a shell account
  4. Use diagnostic tools
    1. View and download system log files
    2. Collecting log files for hypervisor image troubleshooting
    3. Diagnosing Advanced Middleware Configuration problems with message logs
    4. Diagnosing Advanced Middleware Configuration problems with message logs
    5. Enable Advanced Middleware Configuration tracing
    6. Advanced Middleware Configuration logs
    7. Rules for the standard output stream
    8. Location and description of product logs
    9. View product logs
    10. Log customization
    11. Log configuration for the server-side component of the Eclipse client
    12. Log level settings
    13. Global logging component
    14. Customize logging properties for action trace logs
    15. Customize logging properties for framework server trace logs
    16. Customize logging for Jython classes
    17. Customize logging properties for the Tomcat server
    18. Monitor LED status
    19. Monitor system events
    20. Monitor system problems
    21. Manage trace settings
    22. Configure tracing for the workload agent
  5. Troubleshoot by symptom
    1. CMM event occurs when compute node temperatures differ
    2. Troubleshoot LDAP connection issues
    3. Hypervisor errors and scenarios that can cause them
    4. Self-describing capability does not work for an external PureApplication System Agent
    5. Troubleshoot the System Monitoring service
    6. KFWITM088W Additional Topology information not available
    7. No monitoring data for the PureApplication System
    8. TEPS connectivity error
    9. System Monitoring service remains in launching state for over an hour
    10. Error occurs when middleware monitoring shared service is deployed without a running System Monitoring shared service
    11. Refresh a WebSphere Application Server configuration with the proxy shared service
    12. Environment profile not available for virtual application pattern deployment
    13. Virtual systems deploy successfully with failed scripts
    14. Situation events not forwarded to event receivers
  6. Troubleshoot Image Construction and Composition Tool issues
    1. Log files for troubleshooting
    2. Log files for synchronization
    3. Adjusting the configuration timeout settings
    4. Set logging levels for troubleshooting
    5. VMware ESX cloud provider troubleshooting
    6. Troubleshoot problems during VM activation
    7. VMware ESX cloud provider creation fails
    8. Work with enablement bundles
    9. Update Activation Engine in a virtual image
    10. Image Construction and Composition Tool does not respond
    11. Image Construction and Composition Tool does not start
    12. Image Construction and Composition Tool returns EOFException
    13. Install executable launcher error
    14. Install executable error during installation
    15. Package installation error during silent upgrade
    16. Incorrect synchronization parameters after upgrade
    17. Universal ID and version must be unique
    18. Cancel - use the current file link does not work
    19. VMware ESX error if you use virtual machines with snapshots
    20. Failure during OVA disk conversion
    21. Import the same OVA image into multiple VMware ESX cloud providers fails
    22. Import or export of the OVA file fails
    23. Import a virtual machine fails
    24. Error occurs during OVA export
    25. OVA deployment to VMware fails with unsupported element 'Property'
    26. The script package maestro is in failed state
    27. Virtual image capture fails
    28. Virtual images cannot be deployed
    29. Files are not replaced correctly
    30. The user interface stops responding while uploading files
    31. Limitation when performing concurrent updates
    32. Cannot remove bundles from personality
    33. Bundles are not displayed
    34. Limitation when adding bundles to a personality
    35. Bundles not added to new images
    36. Limitation when deleting images that are synchronizing
    37. Time lag when deleting virtual images
    38. SSL exception
    39. Cannot pass characters for arguments
    40. TCP/UDP firewall ports might be appended incorrectly on Red Hat system after synchronization
    41. Virtual image synchronization
    42. Virtual image synchronization takes a long time
    43. Virtual image synchronization fails after five hours for images
    44. Synchronization fails for a virtual image imported from a running virtual machine
    45. Virtual image synchronization when using a trial version of VMware ESX
    46. Virtual image synchronization fails because of return code value
    47. Virtual image synchronization fails and troubleshooting is not possible
    48. Windows image synchronization fails
    49. Cannot synchronize Windows 2008 R2 image
    50. Timeout with Linux bundle in Windows image on VMware
    51. Image synchronization from the command line does not work
  7. Troubleshoot Advanced Middleware Configuration

      7.1. Troubleshoot the generation process
      7.2. Clearing Jython cache
      7.3. Test server connections
      7.4. Connection to UNIX or Linux systems fails with 'Broken pipe' message
      7.5. Troubleshoot installation errors
      7.6. Troubleshoot Workload Deployer errors
      7.7. Missing configuration in console after upgrade
      7.8. Automation plan fails with no output in results
      7.9. Garbled data in .properties file
  8. Rational Automation Framework error messages

      8.1. Error messages overview


Troubleshoot and support

Use troubleshooting information to isolate and determine issues that you might be encountering.

You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps. The steps to troubleshoot an issue vary depending on the problem. To help make relevant information available to you as quickly as possible, the log files and other tools for troubleshooting problems are consolidated. Not all steps are relevant for every problem.

To call IBM Support to resolve an issue, remember to specify the machine type model (MTM) number and the serial number (SN) of the system, rather than those of an individual component. These details are used to connect you with the appropriate IBM service representative who has PureApplication System expertise.

A note about Storehouse Browser: If an issue requires contacting IBM Support, the representative might ask you to locate component or other resource information that is stored in the PureApplication System Storehouse. To find that information, use the Storehouse Browser, which you can access with the system console.

  1. Locate the MTM and SN to provide to the IBM Support. These numbers are displayed together on the top-right corner of the physical system in the following format:
    <MTM_digit> - <SN_digit>, where X is an MTM digit, and Y is an SN digit.

    MTM reference: See the following table to confirm the correct MTM of the system. As column 2 of the table indicates, the only configuration option that differentiates the offerings is the number of compute nodes.

    SNs for MTM 8283 12X-3S2

    MTM-SN Number of compute nodes Location of compute nodes
    8283-12X 2 Chassis 1: Bays 5 and 6
    8283-14X 4 Chassis 1: Bays 5, 6, 7, and 8
    8283-3C2 6
    • Chassis 1: Bays 3 and 4
    • Chassis 2: Bays 3 and 4
    • Chassis 3: Bays 1 and 2

    8283-3F2 12
    • Chassis 1: Bays 3, 4, 5 and 6
    • Chassis 2: Bays 3, 4, 5 and 6
    • Chassis 3: Bays 1, 2, 3 and 4

    8283-3L2 24
    • Chassis 1: Bays 3 - 10
    • Chassis 2: Bays 3 - 10
    • Chassis 3: Bays 1 - 8

    8283-3S2 38
    • Chassis 1: Bays 3 - 14
    • Chassis 2: Bays 3 - 14
    • Chassis 3: Bays 1 - 14

  2. Contact IBM.
    • In the U.S. and Canada, call 1-800-IBM-SERV (1-800-426-7378). Select the second VRU option for software.
    • To locate a global IBM support contact, see the Directory of worldwide contacts at http://www.ibm.com/planetwide/. Select the contact number or VRU option for software.


Enable account access for the IBM service representative

Use the system console to enable access for the IBM service representative to service the system.

You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps.

After a Problem Management Record (PMR) is opened, you are asked to prepare the system for service by an IBM service representative. Use the system console to enable the IBM service representative account, which is used by the IBM service representative to log in to the system console for the purpose of servicing the system. The account remains enabled for 36 hours. It becomes disabled after 36 hours has passed.


Procedure

  1. Access the console pane.
    Click System Console > System > Troubleshooting.

  2. Select the Enable IBM service representative account access check box.

  3. Click Generate to create a secret key. The encrypted symmetric key populates the Secret key field. You must provide this key to IBM to request the account credentials. This key is used by the IBM Service Center for a time-limited password.

  4. Contact IBM. Specify the machine type model (MTM) and the serial number (SN) of the system, rather than those of an individual component.
    • In the U.S. and Canada, call 1-800-IBM-SERV (1-800-426-7378). Select the second VRU option for software.
    • To locate a global IBM support contact, see the Directory of worldwide contacts at http://www.ibm.com/planetwide/. Select the contact number or VRU option for software.


Obtaining a collection set of problem determination information for IBM support

Use the console to obtain a set of problem determination information to assist with debugging problems with the system. IBM support provides you with an IP address to use when completing the steps in this topic.

You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps.


Procedure

  1. Export system events:

    1. Click System Console > System > Events.
    2. Click the Export icon and follow the prompts to save the event information.

  2. Create an image of the infrastructure map:

    1. Click System Console > Hardware > Infrastructure Map.
    2. Save a screen capture of the hardware map.

  3. Obtain the IP collection set. This collection set contains trace and log files that IBM support uses to help diagnose and debug any problems. These size of these files is large, and you must use FTP to provide them to IBM support.

    1. Click System Console > System > Troubleshooting.
    2. Expand the System Log section, and click Collect Log.
    3. The default collection set is Management. Change this setting to IP. Enter the IP address that has been provided by IBM support. Click OK.
    4. When the state of the request changes to Available click the Download icon, and follow the prompts to save the file.

  4. Contact IBM Support to open a Problem Management Record (PMR). For information on contacting IBM support, refer to the instructions provided in Contacting IBM support.

  5. Use the PMR number that IBM support provides, send the problem determination documentation that you have gathered. Follow the steps that are outlined in Exchanging information with IBM PureApplication System technical support.

    Note: Do not send any confidential information from your organization.


Create a shell account

A shell account must be used to access the system's troubleshooting features. After you create the shell account, use the help command to retrieve a list of available troubleshooting commands.

You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps.


Procedure

  1. Use the console to create the shell account. Click System Console > System > Troubleshooting.
  2. Expand Shell account.
  3. Enter a password in the Password of current user field and click Create.



4.1. View and download system log files

The system contains stored log data that you can view directly from the system console, or you can download the information to your local file system.

You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps.

You can view and download two log files: error.log and trace.log. Each log file has a maximum size that it cannot exceed and a file limit defining the number of versions of that file that can be maintained on the system. Each error.log file can be a maximum of 2 MB in size and up to five versions of the file are maintained for a total of 10 MB of available data. Each trace.log file can be a maximum size of 100 MB in size and up to 20 versions of the file are maintained (1 active file and 19 archived files). If a version of either log file is older than 30 days, it is automatically deleted.


Procedure

  1. Complete the following steps to view and download logs from the workload console:

    1. Click System > Troubleshooting.

    2. Choose from the following options:

      Kernel Service log file

      View the log files for kernel services. Kernel services trace activity related to virtual application deployments, not virtual systems.

      Storehouse log file

      View the log file for the Storehouse Browser. The Storehouse Browser is a repository used as a key component for troubleshooting problems. The Storehouse log file contains activity related to Storehouse database access used for virtual application deployments.

      View current error file

      View details about the current list of high level messages (ERROR, WARNING, INFO) from the main component which manages images, patterns, the user interface, and all stages of deployments. By choosing this option, a new browser view opens and streams all new content written to the error file in real time until you close or pause the view.

      View current trace file

      View details about the current trace file. In addition to containing the same content as the error file, the trace file contains all enabled trace levels (FINE, FINER, FINEST, ALL) by component. By choosing this option, a new browser view opens and streams all new content written to the trace file in real time until you close or pause the view.

      Download log files

      Download a trace.zip file that contains all log files, both current and archived.

      When the file size limit is reached, the file is compressed and archived to a *.log.gz file, for example:

      -rw-r--r-- 1 root root  7086206 Jan 11 11:25 trace_2013.01.11_11.25.09.0707.log.gz
      -rw-r--r-- 1 root root  7048265 Jan 11 13:22 trace_2013.01.11_13.22.44.0406.log.gz
      -rw-r--r-- 1 root root 59042938 Jan 11 14:23 trace.log
      
      Then, data is logged to a new, uncompressed file.

      Download the latest log files only

      Download a trace.zip file that contains only the current, uncompressed log files and not the archived files. By choosing this option, you can download a significantly smaller trace.zip file in less amount of time than when you download a trace.zip file that contains both current and archived files.

      Remove old traces and logs

      Delete the archived files for workload console logs only. The currently active files remain available.

  2. Complete the following steps to download a collection of logs from the system console:

    1. Click System > Troubleshooting.

    2. Expand the System Logs section, and click Collect System Logs.

    3. From the Request System Logs panel, select one of the following collection sets, and click OK:

      System

      Collects logs for the system including:

      • Management collection logs
      • Deploy collection logs
      • System configuration (status and inventory).

      Management

      Collects PureApplication System management logs. This collection contains data to help resolve problems with the system console and runtime.

      Deploy

      Collects workload console logs. This collection contains data to help resolve problems with workload management. For diagnostics on any virtual images, use the viewer within the workload console.

      Complete

      Collects the following:

      • System collection set
      • Dumps collection set
      • Logs from the management node

      Dumps

      Collects JVM diagnostic information from the PureSystems. Manager. This collection set contains diagnostic information from the JVM which supports the system console functions. This information is primarily used by the IBM support team.

      IPaddress

      Select the IP collection set to use from the provided list. Diagnostic data is collected from specific components including Chassis Management Module(CMM), IBM Storwize V7000, IBM RackSwitch. G8264, IBM Flex System EN4093 10Gb Ethernet Scalable Switch, IBM Flex System FC5022 16Gb SAN Scalable Switch, management nodes and compute nodes. Complete the following steps to obtain the relevant IP address for the component for which logs are required:

      • Use the following REST API URI to return a list of rack components: http://<IP address>/admin/resources/systemlogs?ipmap
      • Examine the REST response to identify the relevant IP address for the component for which you require logs. Note that all IP addresses that are listed for one particular component are valid.

      For assistance in determining the IP address, contact IBM Support.

      The data is generated in a table format, which includes the following actions that you can choose from:

      Export

      Export a systemlogs.csv file to your local system. The file contains a summary of the log collections already created.

      Refresh

      Regenerates values contained in the table.

      Download

      Download the log collection set to your local system. The generated logs are in compressed archive format (tgz).

      Delete

      Delete the log collection set.


4.2. Collecting log files for hypervisor image troubleshooting

When you create a new hypervisor image, or extend and capture an existing hypervisor image, you can specify which log files or folders to collect for troubleshooting.

The log files or folders that you specify to be collected for troubleshooting can be collected using one or more of the following ways:

To specify which log files or folder should be collected for troubleshooting, perform the following steps:

  1. Create a directory named /etc/logging, if it does not already exist.
  2. Create a JSON file named <component>.json where component is a unique name within /etc/logging to help identify which log files and folders to collect.
  3. Edit the <component>.json file to contain the list of files, folders, or folders with patterns that should be monitored for log files for a specific role, for example, operating system, middleware, or other. The following example provides a listing for a role named "OS" where a specific log file named abc.log, everything recursively in the /opt/logs folder, and only files with the .log file extension are monitored.

    { "role":"OS", 
      "types": [ { 
      		 		 "logtype": "SingleLine", 
      		 		 "type": "file", 
      		 		 "name": "/log/abc.log"
      		 		 }, {
      		 		 "logtype": "File", 
      		 		 "type": "dir", 
      		 		 "name": "/opt/logs"
      		 		 },{
      		 		 "logtype": "File", 
      		 		 "type": "dir", 
      		 		 "name": "/opt/specificlogs",
    				 "pattern":"*.log"
      		 		 }
       ] }
    

    See the following table for a description of the values that you can use for logtype:

    Logtype name Logtype details
    File Single file created for log entry
    BinaryFile Single binary file created for log entry
    SingleLine Each line is a new log entry
    MultiLineTimeStamp Single/Multi line entry where square bracket encased date/time [10/8/10 16:42:54:109 EDT] notes the start of a new entry.

    "start": "\\[\\d{1,2}/\\d{1,2}/\\d{2}.*\\d{1,2}:\\d{2}:\\d{2}:\\d{3}.*\\w{1,3}\\].*"

    MultiLineIP Single/Multi line entry where ip address notes the start of a new entry

    "start": "\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}.*"

  4. To ensure that the log collection is working as expected, run the MustGather script package on a deployed virtual system using the hypervisor image. The output .zip file should contain the logs and folders that you specified if they exist on the deployed system.


4.3. Diagnosing Advanced Middleware Configuration problems with message logs

The topics contained in this section provide information that is intended to help you diagnose and solve problems that you might encounter with Advanced Middleware Configuration using message logs and trace information.


4.4. Monitor LED status

Use the system console to monitor the LED status for hardware in the system.

You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps.


Procedure

  1. Access the console pane. Click System Console > System > Troubleshooting.
  2. Expand LED status. A list of processors is provided in a table, and each processor is associated with a compute node. The table also includes the status of the LEDs.
  3. Filter the data. Select the status of the LEDs to filter the processors that are displayed in the table. In the Status menu, select All, Off or On. By selecting All, you can generate a table that includes all processors that have LED status on or off. By selecting Off, you can generate a table that includes all processors for which LED status is Off. To list processors with the LED status On, select On from the menu.
  4. View the compute node source. Click the <compute_node> link in the Source column of the table.

You filtered the LED status for processors and viewed the compute nodes associated with those processors.


4.5. Monitor system events

You can set filters, such as event type, severity, and category, to refine the list of system events that are displayed. Narrowing the list of events is helpful when you want to troubleshoot issues for a certain device or component in the system.

You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps. You can also access events from the tree view of the infrastructure map. The Infrastructure Map pane default is the graphics view. To ensure that you are on the tree view, select Show Tree View.


Procedure

  1. Click System > Events.

  2. Filter the events.

    1. To filter by event type, select one or more of the following values in the Type field. If you do not specify a filter value, all system events are displayed.

      • Select All
      • Chassis Management Module
      • Chassis Network Switch
      • Chassis Node
      • Chassis Power Module
      • Chassis San Switch
      • Chassis Switch
      • Compute Node
      • LDAP
      • Management Node
      • PureApplication Service
      • PureApplication Systems Manager
      • Storage Subsystem
      • Systemlogs
      • Virtual Machine
      • Virtual Management Instance
      • Virtual Systems Manager
      • Workload Monitoring

      The values are not static. The values are returned from the server side and not restricted to the listed values. Also, in some cases, you might see values that begin with Unknown. These values indicate that the server has either located events from a component that is not yet known or is still displaying events from a component that has been recently replaced but not yet updated on the server.

      When you are viewing events, note the following limitations:

      • New events that are introduced in the current release are not aggregated with existing events from releases earlier than PureApplication System Version 1.0.0.4.
      • The event text is displayed in English only.

    2. Select from the following values in the Severity menu. You can select more than one value. After you select the type and severity, the list is generated automatically.

      Select All

      All event messages generated by the system.

      Fatal

      Event messages that will likely result in an immediate failure of the system.

      Critical

      Critical events require assistance from IBM support. However, make sure to understand the criteria for deciding how urgently to contact IBM support. With some critical events, the machine can continue to operate successfully due to the redundancy of the hardware, the networking and overall architecture. If a critical event occurs repeatedly, which is indicated by the count field, contact IBM support and gather the appropriate logs.

      Major

      Event messages that can or have affected the system, and should be addressed with high priority.

      Minor

      Event messages that can or have affected the system, and can be addressed with lower priority.

      Warning

      Warning events provide useful information within the context of the system, and are often used by IBM support to diagnose problems that are identified by failures, as well as critical and call support events. These events can reflect situations that might eventually lead to a critical event if the automated recovery steps fail. The system should recover, but warning events provide a breadcrumb trail for IBM support if required.

      This type of event represents state changes within the system, but is not necessarily a cause for concern or action. For example, the Image Construction and Composition Tool sends an event when the tool is stopped, which can impact to other users of the system. The system administrator can track this event to identify why the workload was stopped. You can use the user interface as described in step 2 to filter these events and identify more serious events that might be occurring within the system.

      Informational

      Informational events provide useful information within the context of the system, and are often used by IBM support to diagnose problems that are identified by failures, as well as critical and call support events.

      This type of event represents state changes within the system, but is not necessarily a cause for concern or action. For example, the Image Construction and Composition Tool sends an event when the tool is stopped, which can impact to other users of the system. The system administrator can track this event to identify why the workload was stopped. You can use the user interface as described in step 2 to filter these events and identify more serious events that might be occurring within the system.

      Audit

      Events that are the result of system auditing.

      Debug

      Messages that are displayed when other errors are being debugged.

      Unknown

      Event messages from a device that does not have a severity associated with it. These event messages are typically the lowest type of severity.

    3. Select the following category types from the Category menu.

      • All
      • Alert
      • Resolution
      • Call support: Events of this category can have an associated critical severity, which indicate that something has failed and requires urgent attention. Call support events that are not critical represent a piece of hardware that has detected a failure but will continue working for a period of time. Even though attention to these events is less urgent, they can result in a critical failure and the hardware might become inoperable if ignored. As a best practice, still contact IBM support for any Call support event.
      • Customer serviceable: Events of this category can include required physical checks, for example CMM event occurs when compute node temperatures differ. If the physical checks do not uncover an environmental problem that you can resolve, contact IBM support for further assistance with collecting the correct hardware logs and resolving the problem.

    4. Select the following options from the Time interval menu.

      All

      Shows all events

      Last Hour

      Shows the events that occurred in the last hour

      Last 24 Hours

      Shows the events that occurred in the last 24 hours

  3. View the event count. The count displays how many times the event occurred. Click the number to show the detailed timestamp. Click the arrow in the Count column header to show the most frequent event at the top of the list.

    Contact IBM support for assistance with recurring events even if you do not observe failures with workloads or other functions.

  4. Add a comment on an event. You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps. With this permission, you can only comment on events related to hardware.

    1. Click the Comments icon in the Actions column.
    2. Type a comment in the field.
    3. Click Post a comment.
    4. Click OK.

  5. Delete an event from the list. Click the Delete icon in the Actions column. To delete more than one event at the same time, select the check boxes beside the events, and click the red X icon above the Actions column. To delete all events or all events in the current filter, select the check box at the top of the event list next to the Event Text column title. Then click the red X icon.

  6. View the event details. Click the Show details icon in the Actions column. If you open a PMR related to an event, ensure that you include the information from the event details page.


4.6. Monitor system problems

Use the system console Problems pane to monitor and filter problems that occur in IBM PureApplication System W1500.

You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps. This pane lists all of the problems that exist in the system. You can filter the problems by status or failed machine. Included in the table are problem text to describe the problem, the machine that failed, problem details, status such as open and closed, and the date that the problem arose. Any event that is marked as Call support is added to the table. These events represent issues that require assistance from IBM support.


Procedure

  1. Click System > Problems.
  2. Filter the system problems by status. Click the Status menu and select from one of the following: All, Open, or Close.
  3. Filter the status by failed machine. Enter the name of the failed machine in the Failed machine field.
  4. Review the details for each problem by clicking the Show details icon in the Action column.
  5. Export a list of system problems to a file that you can download to your local system. Click the Export icon and Save.
  6. Delete a problem by clicking the Delete icon in the Action column.


4.7. Manage trace settings

As you troubleshoot issues with the system, add or modify trace settings, the details of which you can download in log files, to effectively diagnose and solve problems.

You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps.

You can view and download two log files: error.log and trace.log. Each log file has a maximum size that it cannot exceed and a file limit defining the number of versions of that file that can be maintained on the system. Each error.log file can be a maximum of 2 MB in size and up to five versions of the file are maintained for a total of 10 MB of available data. Each trace.log file can be a maximum size of 100 MB in size and up to 20 versions of the file are maintained (1 active file and 19 archived files). If a version of either log file is older than 30 days, it is automatically deleted.

A set of default classes is defined as the trace string to be included in the logs. The level of trace for these classes can be modified and new classes can be added. The trace levels provided are based on Java. Logging convention and WebSphere Application Server levels.


Procedure

  1. Click System > Troubleshooting.

  2. Expand the Trace Setting section.

  3. Modify the existing trace levels. Click the current trace level setting and select the new trace level from the available options, which are listed as follows:

    OFF

    No events are logged

    SEVERE

    The task cannot continue, but the component can still function

    WARNING

    Potential error or impending error

    INFO

    General information outlining overall task progress

    FINE

    Trace information - General trace

    FINER

    Trace information - Detailed trace + method of entry /exit / return values

    FINEST

    Trace information - A more detailed trace - Includes all the detail that is needed to debug problems

    ALL

    All events are logged. If you create custom levels, ALL includes your custom levels and can provide a more detailed trace than FINEST.

  4. Add new trace settings.

    1. Scroll to the bottom of the list, and click Add trace setting.
    2. Enter the name of the trace setting in the field.
    3. Select the trace level setting. By default, the trace level is set to INFO.
    4. Click OK to save your changes.

  5. Delete trace settings by clicking the X icon.


4.7.1. Configure tracing for the workload agent

You can enable tracing for the workload agent process to assist with debugging and troubleshooting virtual applications. By default, only information level messages are recorded. When you need more detailed trace logs, you can enable all log levels. There are two ways to enable tracing for the workload agent.


Configure tracing for all new deployments

Use these steps to enable tracing for the workload agent in all new virtual application deployments.

Note: When detailed tracing is enabled for the workload agent, the performance of virtual applications might be impacted. When you no longer require tracing to troubleshoot, reset the trace string as described at the end of this procedure.

  1. Click the Workload Console tab at the top of the Welcome page to open the workload console.

  2. Click Cloud > System Plug-ins.

  3. Select a Foundation Pattern Type from the list, and then select the agent plug-in.

  4. Click Configure on the toolbar.

  5. Specify the appropriate settings

    Trace string applied to the agent

    Type the trace string.

    • The default trace string is *=info=enabled. This setting records only info level messages, which are saved in the trace log.

    Use the following string to enable all traces. Enter the trace string as one continuous string without spaces or line breaks.

    *=info=enabled:com.ibm.maestro.common.*=all=enabled:com.ibm.maestro.agent.*=all=enabled:com.ibm.maestro.profile.
    *=all=enabled:com.ibm.maestro.security.*=all=enabled:com.ibm.purescale.profile.*=all=enabled:com.ibm.purescale.security.
    *=all=enabled:osgi=all=enabled:com.ibm.ws.bbson.ConfigOracle=all=enabled:com.ibm.ws.bbson.api.connectivity.local.
    *=all=enabled:com.ibm.ws.bbson.api.traffic.local.*=all=enabled:com.ibm.ws.bbson.api.connectivity.remote.
    *=all=enabled:com.ibm.ws.bbson.api.traffic.remote.*=all=enabled:com.ibm.ws.bbson.BBFactoryImpl
    =all=enabled:com.ibm.maestro.elbservice.management.*=all=enabled:com.ibm.maestro.sharedservice.genericREST.resources.
    *=all=enabled
    

    Trace File Name

    The name for the trace file. The default file name is trace.log.

    Trace Max File Size

    The maximum trace log file size in MB. The default size is 40 MB. When a log file reaches the specified maximum size, a new log file is created. The file name for the new trace log includes a numeric suffix, and the suffix increments for each new file. For example, trace.log.1, trace.log.2, trace.log.3.

    Trace Max File Number

    The maximum number of trace log files that can be created. The default is 20 files. Each trace log file has a numeric suffix. For example, trace.log.0, trace.log.1, trace.log.2, trace.log.3. If the maximum number of files is reached, a new sequentially numbered trace log file is created and the oldest trace log file is deleted. For example, if the maximum value is 20, only 20 trace log files are saved at a time. The file trace.log.20 is created when trace.log.19 reaches the maximum file size and trace.log.0 is deleted.

    Append to trace file

    If this option is selected, data is appended to the most recent existing trace log file each time a new set of information is recorded to the trace log until the maximum file size is reached. If this option is not selected, a new trace log file is created each time a new set of information is required in the trace log, even if the maximum file size of the most recent log file is not reached.

  6. Click Update. Tracing is enabled for all the workload agents in new deployments.

  7. After you deploy a virtual application, you can view the trace logs for the agent. Perform the following steps.

    1. Click Instances > Virtual Applications.

    2. Select a virtual application instance.

    3. Click Manage on the toolbar.
    4. Click the Logging tab. Logs are organized by virtual machine name, and each name contains the name of the roles that are associated with the virtual machine. A role within a virtual application refers to the software that is installed and configured.
    5. Expand the IWD Agent role and then expand the entry with the /logs subdirectory. For example, if the virtual machine name is Web_Application-was.11332975226124, expand the entry Web_Application-was.11332975226124/logs.

      The log file with the file name that you configured is listed under the /logs directory. Select the trace log to view the contents in the detail pane.

  8. When you no longer need the trace logs for troubleshooting or debugging, reset the trace string. Repeat the preceding steps and set the trace string as follows:

    *=info=enabled
    


Configure tracing for a specific virtual application instance

If you need detailed trace logs for the workload agent in an existing virtual application instance and tracing of all log levels was not configured at deployment time, you can use these steps to configure detailed tracing for the workload agent in the virtual application instance.

Note: When detailed tracing is enabled for the workload agent, the performance of virtual applications might be impacted. When you no longer require tracing to troubleshoot, reset the trace string as described at the end of this procedure.

  1. Click the Workload Console tab at the top of the Welcome page to open the workload console.

  2. Click Instances > Virtual Applications.

  3. Select a virtual application instance.

  4. Click the Manage icon on the toolbar.

  5. Click the Operation tab.

  6. Select the Agent role from the list operations. The detail pane displays configuration options.

  7. Expand Update trace settings and specify the appropriate settings.

    Trace string applied to the agent

    Type the trace string.

    • The default trace string is *=info=enabled. This setting records only info level messages, which are saved in the trace log.

    Use the following string to enable all traces. Enter the trace string as one continuous string without spaces or line breaks.

    *=info=enabled:com.ibm.maestro.common.*=all=enabled:com.ibm.maestro.agent.*=all=enabled:com.ibm.maestro.profile.
    *=all=enabled:com.ibm.maestro.security.*=all=enabled:com.ibm.purescale.profile.*=all=enabled:com.ibm.purescale.security.
    *=all=enabled:osgi=all=enabled:com.ibm.ws.bbson.ConfigOracle=all=enabled:com.ibm.ws.bbson.api.connectivity.local.
    *=all=enabled:com.ibm.ws.bbson.api.traffic.local.*=all=enabled:com.ibm.ws.bbson.api.connectivity.remote.
    *=all=enabled:com.ibm.ws.bbson.api.traffic.remote.*=all=enabled:com.ibm.ws.bbson.BBFactoryImpl
    =all=enabled:com.ibm.maestro.elbservice.management.*=all=enabled:com.ibm.maestro.sharedservice.genericREST.resources.
    *=all=enabled
    

    Trace File Name

    The name for the trace file. The default file name is trace.log.

    Trace Max File Size

    The maximum trace log file size in MB. The default size is 40 MB. When a log file reaches the specified maximum size, a new log file is created. The file name for the new trace log includes a numeric suffix, and the suffix increments for each new file. For example, trace.log.1, trace.log.2, trace.log.3.

    Trace Max File Number

    The maximum number of trace log files that can be created. The default is 20 files. Each trace log file has a numeric suffix. For example, trace.log.0, trace.log.1, trace.log.2, trace.log.3. If the maximum number of files is reached, a new sequentially numbered trace log file is created and the oldest trace log file is deleted. For example, if the maximum value is 20, only 20 trace log files are saved at a time. The file trace.log.20 is created when trace.log.19 reaches the maximum file size and trace.log.0 is deleted.

    Append to trace file

    If this option is selected, data is appended to the most recent existing trace log file each time a new set of information is recorded to the trace log until the maximum file size is reached. If this option is not selected, a new trace log file is created each time a new set of information is required in the trace log, even if the maximum file size of the most recent log file is not reached.

  8. Click Submit. Tracing is configured for all the workload agents in the deployment.

  9. Use the following steps to view the trace logs for the agent.

    1. Click the Logging tab. Logs are organized by virtual machine name, and each name contains the name of the roles that are associated with the virtual machine. A role within a virtual application refers to the software that is installed and configured.

    2. Expand the IWD Agent role and then expand the entry with the /logs subdirectory. For example, if the virtual machine name is Web_Application-was.11332975226124, expand the entry Web_Application-was.11332975226124/logs.

      The log file with the file name that you configured is listed under the /logs directory. Select the trace log to view the contents in the detail pane.

  10. When you no longer need the trace logs for troubleshooting or debugging, reset the trace string. Repeat the preceding steps and set the trace string as follows:

    *=info=enabled
    


Troubleshoot by symptom

Troubleshoot the system by symptom when there is limited event or error code information or when you observe problems that are not reflected in the system logs.


5.1. CMM event occurs when compute node temperatures differ

A warning message is displayed when the values between the compute nodes that report the highest and lowest temperatures differ by 5 degrees Celsius.


Symptoms

When there is a difference of 5 degrees Celsius between the compute node that reports the highest temperature and the compute node that reports the lowest temperature for a particular chassis, the following warning message is displayed in the events log:

40050000: Hot air exiting from the rear of the chassis might be recirculated in the inlet air at the front of the chassis
Common causes for this event are due to either cold or hot air escaping from beneath the system and reaching the lowest ambient temperature sensors. In some cases, the current system might be properly sealed, but the adjacent system can have openings that allow hot air from the exhaust aisle to recirculate to the front of the system.


Resolving the problem


5.2. Troubleshoot LDAP connection issues

Use this information to troubleshoot possible Lightweight Directory Access Protocol (LDAP) connection issues.

During the LDAP authentication process, general authentication or internal authentication failures might occur, that can prevent a successful login. General authentication failures are due to incorrect user name and password entries. Internal authentication failures can be caused by any of the following errors:

In any case, the following message is displayed:

Your user name or password is not valid.
When a user reports that they cannot log in to the system, even though they are typing the correct password, the administrator can review system events to find out what is causing the authentication issue.

Note: Numerous types of events might be recorded in the system log. To only view LDAP events, sort them by changing the Type field to LDAP. If LDAP does not appear in the list, that means that no LDAP connection issues were detected. The following is an example of what might be displayed in the log:

CWZIP4665W: The connection to LDAP has failed. The following error occurred: CommunicationException: 172.16.248.10:389

This message communicates that an error occurred during a connection attempt to the LDAP server, which might be due to an incorrect PureApplication System login. It can also be caused by a mis-configured parameter under the LDAP settings tab. To troubleshoot LDAP connection failures, complete the following procedure:


Procedure

  1. Review the event details for the exception that was caught. To access the system event logs, click System > Events. When you receive an internal error during LDAP authentication, an event entry is created of the type .LDAP.. The following example shows a possible exception:

    CommunicationException: Connection Timed Out
    

  2. Ensure that the LDAP settings are correct. To verify, click System > Security.

    Note: If a login failure is reported, and the event log does not contain an entry specifying that the connection to the LDAP server has failed, then the log in failure is more likely to be a general authentication issue.

A resolution event entry is created on the Events page, that informs you that the connection to the LDAP server has been restored, following an internal error with the LDAP server. The event is generated only once a successful request to the LDAP server has been made. You might see a message similar to the following example:

CWZIP4666I: The connection to LDAP has been restored.


5.3. Hypervisor errors and scenarios that can cause them

You can receive error messages for hypervisors defined to the system under certain circumstances.

If you change a password on a started hypervisor, the system gives you error messages. The following error messages are shown for each of the following scenarios:

Login is not valid

This message is shown, in the user interface, on the Hypervisors window if you have performed the following steps:

  1. Defined a hypervisor to the system and started it.
  2. Changed the password on the hypervisor, but did not change it in the system.
  3. Rediscovered the hypervisor in the system.

Be sure that you change the password for the hypervisor on PureApplication System, and not just on the hypervisor itself, to prevent this issue.

Login is not valid

This message is shown, in the user interface, on the Hypervisors window and also on the deployment wizard if you have performed the following steps:

  1. Defined a hypervisor to the system and started it.
  2. Changed the password on the hypervisor, but did not change it in the system.
  3. Deployed work to that hypervisor, usually by creating a cloud group with just that hypervisor in it.

Be sure that you change the password for the hypervisor on the system, and not just on the hypervisor itself, to prevent this issue.

Unable to connect to endpoint. URL: <address_of_hypervisor_that_cannot_be_reached>

This message is shown, in the user interface, if you have performed the following steps:

  1. Defined a hypervisor to the system and started it.
  2. Disconnected the hypervisor from the network.
  3. Rediscovered the hypervisor in the system.

No physical machines are found for the hypervisor

When you receive this message, you might need to reconnect to the VMware Virtual Center. Use the following general procedure:

  1. If the hypervisors have active deployments, quiesce the hypervisors.
  2. Put the hypervisors in maintenance mode.
  3. Rediscover the VMware Virtual Center.
  4. Re-configure any network or storage options, if they changed because of the rediscover process.
  5. Start the hypervisors.

If the problem is not resolved, you might need to reboot the hypervisors and then reboot and rediscover the VMware Virtual Center.


5.4. Self-describing capability does not work for an external PureApplication System Agent


Problem: Self-describing capability does not work for an external PureApplication System Agent. When the PureApplication System Agent is installed or upgraded, its application support is not automatically installed or upgraded on the remote monitoring server and hub monitoring server.


Solution: Install the IBM Tivoli Monitoring agent framework V6.2.3 or later on the system where the external PureApplication System Agent is installed.


5.5. Troubleshoot the System Monitoring service

This section contains problems that you might encounter while using the System Monitoring service and solutions that you can use to fix these problems. Use the following information to help you troubleshoot problems related to the System Monitoring service:


5.5.1. KFWITM088W Additional Topology information not available

When opening the PureApplication System Monitoring Portal, the error message "KFWITM088W Additional Topology information not available at this time. The network configuration might be changing at this time. Please try again later." is displayed.

To resolve this problem, restart the Tivoli Enterprise Portal Server.


5.5.2. No monitoring data for the PureApplication System

No monitoring data for the PureApplication System is presented in the workspaces of Tivoli Enterprise Portal.

No monitoring data for the PureApplication System is presented in the workspaces of Tivoli Enterprise Portal. This problem occurs if, for some reason, the PureApplication System is down for over five days and then restarted.

To fix this problem, restart the external PureApplication System Agent ...s:

  1. cd ITM_install_dir/bin
  2. ./itmcmd agent -o agentinstance stop q8
  3. ./itmcmd agent -o agentinstance start q8

where ITM_install_dir is the installation directory of IBM Tivoli Monitoring, and agentinstance is the name of the external PureApplication System Agent instance.


5.5.3. TEPS connectivity error

TEPS connectivity error is displayed in the Summarization and Pruning agent workspace.

The value of the TEPS Connectivity attribute in the Connectivity table of the Summarization and Pruning Agent workspace is No. It will cause historical data to be unavailable in the workspaces.

To resolve this problem, restart the Tivoli Enterprise Portal Server.


5.5.4. System Monitoring service remains in launching state for over an hour

The System Monitoring service might remain in launch state after deployment.

Administrators might notice that deployments that are monitored by System Monitoring service have unknown status. During this time, up or down scaling might stop if these deployments use a scaling policy.

As a workaround, if the System Monitoring service has been in launching state for over an hour, delete and redeploy the service. Deployments that have been monitored by the System Monitoring service will reconnect to the newly deployed service, resulting in known health status. No user intervention or down time for the monitored deployments is expected.


5.5.5. Error occurs when middleware monitoring shared service is deployed without a running System Monitoring shared service

Monitor services for middleware depend on the base System Monitoring service to function.

Administrators can extend the System Monitoring service to collect performance and availability data about middleware applications in a cloud group. Monitoring services for middleware are separate shared services deployed in addition to the base System Monitoring service. Examples of these service include monitoring services for IBM WebSphere Application Server and IBM HTTP Servers.

Monitor services for middleware depend on the base System Monitoring service to function. If deployments that are monitored by these services for middleware fail, confirm that the base System Monitoring service is running. If not, deploy the base System Monitoring service. Alternatively, delete the middleware monitoring service. Then try to launch your deployment again.


5.6. Manually refreshing WebSphere Application Server configuration with the proxy shared service

If an IBM WebSphere Application Server instance fails to register with the elastic load balancer a deployment or update of a virtual application instance, you can manually refresh the configuration. If the Secure Sockets Layer (SSL) certificate for the WebSphere Application Server instance is updated, the elastic load balancer also updates the certificate in the storehouse.


Procedure

  1. Click the System Console tab at the top of the Welcome page to open the system console.

  2. Click Instances > Virtual Applications.

  3. Select the virtual application instance. The virtual application instance details display.

  4. On the details pane, check the current status of WebSphere Application Server in the Middleware perspective section. If WebSphere Application Server is in a failed state, you can restart the instance.

  5. Click Manage on the menu. The Virtual Machine Monitoring pane displays.

  6. Click the Operation tab. The Operation pane displays.

  7. To recover a failed WebSphere Application Server instance, select it in the list. Under Fundamental settings, expand Start or restart the server and click Submit.

  8. Select the ELBClient instance. The instance details display.

  9. Expand Refresh Configuration with Proxy Shared Service.

  10. Click Submit. The elastic load balancer tries to register WebSphere Application Server instances in the virtual application instance.

    • If a WebSphere Application Server instance registers successfully to the elastic load balancer, and the SSL certificate file is not updated, a message is displayed to inform you that a certificate update will not be done.
    • If a WebSphere Application Server instance registers successfully to the elastic load balancer, and the certificate file is updated, the certificate file is updated in the storehouse.
    • If WebSphere Application Server fails register to the elastic load balancer, the WebSphere Application Server registration configuration is refreshed. Before refreshing, the WebSphere Application Server role was in a FAILED state. After successfully refreshing, the role is in RUNNING state.

WebSphere Application Server instances in the virtual application instance are registered and any updated certificates are stored in the storehouse.


5.7. Environment profile not available for virtual application pattern deployment

When deploying a virtual application pattern, the required environment profile is not offered for selection.

When deploying a virtual application pattern, the environment profile to use might not be displayed for selection. This might occur if the environment profile is configured to use Pattern deployer to obtain IP addresses, instead of IP Groups. Configuring the environment profile to use Pattern deployer is not supported for virtual application patterns.

This problem might also occur if you attempt an automated deployment, specifying an environment profile and IP group. In this case you might encounter the following error message:

IOError: CMPUT0087E: IP source of Environment profile with ID 7 is not provided by IP group.

To resolve this problem, modify the configuration of the environment profile to use IP Groups instead of Pattern deployer, and try the deployment again.

For information on configuring the environment profile to specify where IP addresses are to be obtained, see Modifying environment profiles.


5.8. Virtual systems deploy successfully with failed scripts

When you deploy a virtual system, failures with the associated deployment scripts can occur.


Symptoms

The deployment status of a virtual system might be displayed as completing successfully even though one or more non-critical scripts associated with the deployment fail to complete successfully.


Resolving the problem

Be sure to examine the History section of the Virtual System Instance page to ensure that associated scripts completed successfully during deployment. To view the historical details, click Workload Console > Instances > Virtual System Instances > instance_name.

Some script packages that are defined to run at deployment time are critical to the success of a deployment, and can cause the deployment to fail if the script does not complete successfully. There are a few scripts, however, that will not result in a failed deployment if they fail during deployment:

Each script execution has an associated script status. After the script has completed execution, the script status will have one of the following values:

RM01012 - COMPLETE

Completed successfully with a return code of 0.

RM01013 - FAILED

Completed with a return code other than 0 or 2, or did not get any result back from the virtual machine,

RM01014 - FAILED

Script could not be transferred to the virtual machine.

RM01015 - FAILED

Script did not complete before the time out expired.

RM01059 - WARNING

Completed with a return code of 2.


5.9. Situation events not forwarded to event receivers

If situation events are not forwarded to event receivers, use the information in this topic to solve the problem.


Problem: A situation is fired and an alert icon is displayed on IBM PureApplication System Monitoring Portal, however; the situation event is not forwarded to event receivers.


Solution: This problem occurs if the situation is not configured to forward events to an event receiver, or if the situation is a customized predefined situation, or if the situation is newly created. To fix this problem, you must ensure that forwarding events to an event receiver is enabled for this situation, and then manually associate the situation with the event receivers. You can do it through IBM PureApplication System Monitoring Portal. You can also do it by exporting the situation, changing its definition, and then importing the situation back to the server.

To fix this problem through the IBM PureApplication System Monitoring Portal, do the following steps:

  1. In the IBM PureApplication System Monitoring Portal, click Situation Editor in the main toolbar.
  2. Select the situation in problem.
  3. Click the EIF tab.
  4. Ensure that the Forward Events to an EIF Receiver option is selected.
  5. Select the event receivers to which you want the situation events to be forwarded from the Available EIF Receivers and then click the left arrow to move them to the Assigned EIF Receivers list.
  6. Click OK to save the situation definition with your changes and close the situation editor.

You can also fix this problem by exporting the situation, changing its definition and then importing it back to the server. Here we assume that the situation in problem is named test_tec1 and its definition is as below:

Name : test_tec1
Full Name :
Description :
Type : Windows OS
Formula : *IF *VALUE NT_Cache.Copy_Read_Hits_% *EQ 1
Sampling Interval : 0/0:15:0
Run At Start Up : Yes
Distribution :
Text :
Action Location : Agent Action Selection : System Command
System Command : *NONE
True For Multiple Items: Action on First Item only TEC Severity : Critical
TEC Forwarding : Y
TEC Destination : 0

Do the following steps to fix this problem:

  1. Export the situation to an XML file:

    C:\ODI>tacmd viewsit -s test_tec1 -e c:\test_tec1.xml
    
  2. Edit the XML file and change the following lines accordingly, to specify a null destination server:

    <SITINFO>
    <![CDATA[SEV=Critical;TFWD=Y;~;"]] >
    </SITINFO>
    
  3. Delete the original situation and import the situation back to the server:

    tacmd createsit -i c:\test_tec1.xml
    


Troubleshoot issues with Image Construction and Composition Tool

Use the information and content included here to understand how issues occur during Image Construction and Composition Tool processes.


6.1. Log files for troubleshooting

Check the Image Construction and Composition Tool log files to determine where problems occurred.

To access the Image Construction and Composition Tool server log files, from the Administer menu, click Download logs. Save the compressed file and examine the log files. The compressed file contains the following server log files:

To access the log files from the logs directory, go to /drouter/ramdisk2/mnt/raid-volume/raid0/logs and review the contents of the trace and error directories.

If problems occur during Image Construction and Composition Tool installation, check the following Installation Manager log files to troubleshoot installation problems:

/var/ibm/InstallationManager/logs/<date_time>.xml
/var/ibm/InstallationManager/logs/ant/<date_time>.log
where <date_time> is the date and time in a format similar to 20111024_1201.

To review specific Image Construction and Composition Tool messages, search in the log files for the lines that start with the message code CYO.


6.2. Log files for virtual image synchronization

If you are synchronizing a virtual image and the deployed VM for the virtual image is created, but the synchronization fails, you can download log files for the synchronization. If the synchronization fails before the deployed VM is created, no log files are available. If the synchronization failed after the deployed VM was created, but before the bundle installation, the following log files are available:

If the synchronization fails during or after bundle installation, the following log files are available:

error.log

The most recent error file

trace.log

The most recent trace file

err.log

The error output of the synchronization script run on the VM, including output of any bundle installation script run during the sync process

out.log

The standard output of the synchronization script run on the VM, including output of any bundle installation script run during the sync process
The err.log and out.log files are the log files for the bundle installation during the synchronization process. They are located in the directory:/drouter/ramdisk2/mnt/raid-volume/raid0/local/images/asset_id/logs where asset_id is the asset identifier for the image. To download the log files, complete the following steps:


Procedure

  1. From the Images screen and expand the Virtual System section of the image that you are synchronizing.
  2. Click Download logs. The compressed file containing the log files is downloaded.


6.3. Adjusting the configuration timeout settings

After you change the configuration timeout settings, you must restart the Image Construction and Composition Tool. You can change the following timeout settings:

     # Configurable timeouts
    "ShimTimeoutInMinutes" : 120,
    
    # Enable Single sign-on
    "EnableSSO" : false,
    
    # ICCT instance user     "InstanceUser": "",
    
    # When executing windows commands how long do we wait before we timeout 
    # in minutes.  This default value for this timeout is 3 minutes.
    "WindowsCommandTimeOut": 3,
 
    # When running the execution bundle during a synchronization,
    # how long do we wait before timing out in minutes.  Default     # values is 6 hours.
    "BundleExeuctionTimeout": 360,
 
    # IWD REST timeout     "restReadTimeout": 90000,
    
    # IWD Max deploy wait time (minutes)  Default value is 0 which 
    # disables the max Deploy time.  If set by default we have put in     # 300 minutes (5 hours).  If this is set it will work in conjunction
    # with the following configuration parameters:
    #      IWD_SYSTEM_CREATION_ATTEMPTS
    #      IWD_WAIT_FOR_SYSTEM_CREATION_RETRY_DELAY
    #      IWD_WAIT_FOR_DEPLOY_STATUS_DELAY
    #      IWD_WAIT_FOR_STARTUP_COMPLETE_DEALY
    #      IWD_MAX_SYSTEM_ACTIVATION_WAIT_TIME    
    "maxIWDDeployWaitTime": 0,     
    # IWD Max number of creation attempts also the same     # number of attempts for deployment status checks.
    "IWD_SYSTEM_CREATION_ATTEMPTS": 120,
 
    # IWD delay between creation retries (seconds)
    "IWD_WAIT_FOR_SYSTEM_CREATION_RETRY_DELAY": 30,
 
    # IWD delay between checks for deployment status in (seconds)
    "IWD_WAIT_FOR_DEPLOY_STATUS_DELAY": 10,
 
    # IWD wait for startup complete (seconds)
    "IWD_WAIT_FOR_STARTUP_COMPLETE_DEALY": 30,
 
    # Amount of time to wait for the system to become active.  This is checked
    # after system creation and deploy status checks.  Default value is 300 
    # minutes. 
    "IWD_MAX_SYSTEM_ACTIVATION_WAIT_TIME": 300,
 
    # How many times to check to see if the capture of an image     # has completed before we fail the capture.  The default     # value is 180 times.
    "IWD_MAX_CAPUTRUE_ATTEMPTS": 180,
 
    # When capturing an images in IWD.  How long to wait before     # we check the status again in seconds.  Default value is 60 seconds.
    "IWD_WAIT_FOR_CAPTURE_STATUS_DELAY": 60,
    
    # Maximum number of time to attempt to capture a VM.
    # Default values is 240 times.
    "SCE_MAX_CAPTURE_VM_ATTEMPTS" : 240,
    
    # Time to wait between connection attempts in seconds.
    # Default value is 30 seconds.
    "SCE_WAIT_BETWEEN_CAPTURE_ATTEMPTS" : 30,
    
    # Maximum number of time to try to connect to a VM.
    # Default value is 120 times.
    "SCE_MAX_CONNECTION_ATTEMPTS" : 120,
    
    # Time to wait between connecting to the VM.
    # Default value is 10 seconds.
    "SCE_WAIT_BETWEEN_CONNECTION_ATTEMPTS" : 10,
    
    # When running commands on the VM.  Time to delay between checking the return
    # code of the command to see if it is done or not.  Default value is 60 seconds     "SCE_WAIT_COMMAND_RETRY_DEPLAY" : 60,
    
    # Maximum number of time to try and delete a VM.
    # Default value is 30 times.
    "SCE_MAX_DELETE_VM_ATTEMPTS" : 30,
    
    # Time to wait between checking to see if the VM is deleted.
    # Default value is 60 seconds     "SCE_WAIT_DELETE_VM_DELAY" : 60,
    
    # Time to wait after the VM's deletion state goes to deprovision.
    # Default value is 30 seconds.
    "SCE_WAIT_DEPROVISION_DELAY" : 30,


6.4. Set logging levels for troubleshooting

When troubleshooting is required, trace logs and error logs are generated. The logged information that is generated depends on definitions that follow the Java Logging convention and WebSphere Application Server levels.

The following log level values, listed from the least amount of logging (OFF) to the most amount of logging (ALL) are available:

To modify the data that is logged, use one of the following methods.


Use the command-line interface

If you use the command-line interface, you can change the logging levels dynamically. The changes take effect immediately and do not require Image Construction and Composition Tool to be stopped and started. However, these changes are lost after stopping the Image Construction and Composition Tool server. For more information, see the following commands:

icct.trace.spec()
icct.trace.set('com.ibm.cloud.icn.execution', 'FINER')


Modify Image Construction and Composition Tool file manually

CAUTION:

Follow the exact syntax and format when editing this file.

You can update the file <install_dir>/config/logging-default.config manually. These changes will take effect the next time the tool is restarted. To update the file, complete the following steps:

  1. Change to the directory <install_dir>. For example:

    cd /opt/IBM/icct
    
  2. Create a backup of the following file:

    ./icn.app/config/logging-default.config
    
  3. Update the following file according to your requirements:

    ./icn.app/config/logging-default.config
    
  4. Stop and restart Image Construction and Composition Tool:

    ./stop.sh
    ./start.sh
    


6.5. VMware ESX cloud provider troubleshooting

You can set a number of logging level values to troubleshoot problems with the VMware ESX cloud provider.

Set the following values:

"com.ibm.cloud.icn.cloudprovider.localesx": "FINEST"
"com.ibm.cloud.icn.core": "FINEST"
"com.ibm.cloud.icn.execution": "FINEST"
"com.ibm.cloud.icn.util": "FINEST"
"com.ibm.rainmaker.vmsupport": "FINEST"
"com.ibm.venture": "FINEST"
"com.ibm.vespa": "FINEST"


6.6. Troubleshoot problems during VM activation

If you encounter problems when activating a VM during deployment or synchronization.

Symptoms

Problems occur when activating a VM during deployment or synchronization.

Resolving the problem

Linux

  1. Log on to the VMware vSphere Client console to access the VM.
  2. Review any log files with the extension .out and .err in the directory /opt/ibm/ae/AR for errors.
  3. In the directory /opt/ibm/ae/AP, check if a file named noap exists. If it does, complete the following actions:

    1. Delete the file noap.
    2. Remove all contents of the directory AR.
    3. Reboot the system.
    4. Check that the VM now activates successfully.

If the problem still persists, complete the following steps:

  1. Check the console as it starts for any indication of failure. In the user interface, you might have to press the Esc key.
  2. Log in and investigate each activation script, activating each class one at a time. The scripts that are run during activation are listed in the file /opt/ibm/ae/AL/master.al.

    For example, if ConfigLocale causes problems, run the following command once:

    AE.sh -s ConfigLocale
    
    If a class fails, run the script command. The file master.al lists which script fails. They are usually in the directory /opt/ibm/ae/AS. Then debug the script, until you find out which line in the script is failing. The next step is to determine why the line in the script failed.
  3. Enter the following command:

    df -k
    
    and ensure that the file activation.iso is mounted, usually in the directory /tmp.

Windows

  1. Log on to the VMware vSphere Client console to access the VM.
  2. Review any log files with the extension .out and .err in the directory c:\windows\setup\ibm\ar for errors.
  3. In the directory c:\windows\setup\ibm\ap, check if a file named noap exists. If it does, complete the following actions:

    1. Delete the file noap.
    2. Remove all contents of the directory ar.
    3. Reboot the system.
    4. Check that the VM now activates successfully.

If the problem still persists, complete the following steps:

  1. Check the console as it starts for any indication of failure. In the user interface, you might have to press the Esc key.
  2. Log in and investigate each activation script, activating each class one at a time. The scripts that are run during activation are listed in the file c:\windows\setup\ibm\al\master.al.

    For example, if ConfigLocale causes problems, run the following command once:

    AE.bat -s ConfigLocale
    
    If a class fails, run the script command. The file master.al lists which script fails. They are usually in the directory c:\windows\setup\ibm\as. Then debug the script, until you find out which line in the script is failing. The next step is to determine why the line in the script failed.
  3. Enter the following command:

    df -k
    
    and ensure that the file activation.iso is mounted, usually CD-ROM.


6.7. VMware ESX cloud provider creation fails

Failure during a VMware ESX cloud provider creation.

Symptoms

When you attempt to create the cloud provider, it fails with message: CYOCP0016E: Unable to read cloud provider data specification.

Resolving the problem

The following issues might apply:


6.8. Work with enablement bundles

Enablement bundles are bundles that hide cloud-specific details. They provide a base on which you can build virtual images using cloud-independent bundles. When you are building a virtual image, Image Construction and Composition Tool automatically selects an enablement bundle for the virtual image that matches your cloud provider and operating system and version.

The Image Construction and Composition Tool automatically selects an enablement bundle for the virtual image. During the virtual image building process, you cannot remove the enablement bundle from the virtual image. If you are creating a virtual image and the base virtual image already contains an enablement bundle, no action is performed.

When you work with virtual images and bundles, you must keep the enablement bundles in Image Construction and Composition Tool workspace. When you select a base virtual image, it is automatically added to the appropriate enablement bundle. After you extend the virtual image, giving it a name, symbolic name, version, and description, you see the following message in the Operating System section of the virtual image:

Activation Framework: <name of enablement bundle selected by the tool>
You can then add bundles to the virtual image and synchronize it.

Do not delete the Enablement Bundle for Exploiter Name enablement bundle. This bundle is required.

If the enablement bundle is removed or not available, the following information is displayed in the Operating System section of the virtual image:

Activation Framework: <Not found>

If you accidentally delete this enablement bundle, you can import it again from the following location: <installation_path>/icn.app/private/expanded/ibm/icn.enablement-<version>/bundles/R1/com.ibm.icon.enablement.ux.iwd_<version>.ras where <installation path> is similar to /opt/IBM/icon. Access this top-level directory and change directory to the appropriate cloud provider to find the enablement bundle for the particular operating system you require.


6.9. Update Activation Engine in a virtual image

If Image Construction and Composition Tool can successfully deploy a virtual image, Activation Engine is automatically updated. However, you might need to update Activation Engine if the image cannot be successfully deployed by the cloud provider.

There is a script provided with the Image Construction and Composition Tool that you can run to update Activation Engine in virtual images.

Note: This topic describes upgrading from Activation Engine version 2.0 and higher. Upgrading from version 1.x is a more complex procedure that requires more than a simple version upgrade. For more information, see the Activation Engine documentation.

You can check the version of Activation Engine in one of the following ways:

Some Activation Engine versions between versions 2.1 and 2.1-1.23 delete user data because some RPM scripts are incorrectly called during the upgrade. So, for versions between 2.1 and 2.1-1.23, invoke the RPM with the --noscripts flag to preserve user data.


Automated upgrade

To perform an automated upgrade, run the following script:/opt/ibm/ae/upgradae.sh new-rpm-path where new-rpm-path is the new RPM that upgrades an installed RPM.

Note: Does not apply to IBM i.


Manual upgrade

To upgrade Activation Engine manually, determine the level of the RPM installed. Depending on the level, run the following command:

RPM prior to version 2.1-1.24

rpm --noscripts --force -Uvh new-rpm-path

RPM from 2.1-1.24 and higher

rpm -Uvh new-rpm-path

Note: Does not apply to IBM i.


6.10. Image Construction and Composition Tool does not respond

Determine the reasons that the Image Construction and Composition Tool might not respond.


6.10.1. Image Construction and Composition Tool does not start after installation

Before running the IBM Installation Manager installer to install Image Construction and Composition Tool on a system, ensure that the port number 443 is not in use by another application. If the port number 443 is not available, Image Construction and Composition Tool does not start after you completed the installation.

Symptoms

The Image Construction and Composition Tool fails to start and a ZSO error occurs.

Causes

Port 443 is in use by another application.

Diagnosing the problem

Check the following Installation Manager log files to troubleshoot the problem:

/var/ibm/InstallationManager/logs/<date_time>.xml
/var/ibm/InstallationManager/logs/ant/<date_time>.log
where date_time is the date and time in a format similar to 20111024_1201.

The following error message appears in the Installation Manager log files:

CWPZC8028E: An unknown ZSO error occurred.
Launching of the ZSO failed with return code -1
CWPZT0601E: Error: Command start failed

Resolving the problem

Before installing Image Construction and Composition Tool, ensure that port number 443 is not in use by another application on the system.


6.10.2. Image Construction and Composition Tool returns EOFException

While running the Image Construction and Composition Tool on a system, you receive a 500 error code and an EOFException.

Symptoms

The Image Construction and Composition Tool fails to respond to user requests and EOFExceptions are generated.

Causes

The Image Construction and Composition Tool has run out of disk space.

Diagnosing the problem

The following error appears in the user interface: java.lang.RuntimeException: Unable to create /drouter/ramdisk2/mnt/raid-volume/raid0/.zero/userzoneStore/.

Check the log file to troubleshoot the problem. The following error message appears in the log file:

[2012-06-19 12:42:16:980 EDT] 00000021 JavaGCSeriali E 
           zero.core.context.zones.JavaGCSerializer 
deserialize CWPZC0029E: Unable to serialize key <[B@7ee67ee6> in the ser zone.
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully
          (ObjectInputStream.java:2298)

Resolving the problem

To resolve the problem, complete the following steps:

  1. Stop the Image Construction and Composition Tool using stop.sh.
  2. Delete all .ser files located in the /drouter/ramdisk2/mnt/raid-volume/raid0/.zero/userzoneStore directory and subdirectories.
  3. Start the Image Construction and Composition Tool using start.sh.
  4. Free up /drouter disk space by deleting any images that you do not need.


6.11. Install executable launcher error during installation

If you try to install the Image Construction and Composition Tool on a system where it already exists, the installation is not successful.

Symptoms

When you run ./install, the Image Construction and Composition Tool installation fails with the following error message:

The Install executable launcher was unable to locate its companion shared library.

Causes

The error indicates that a version of the Image Construction and Composition Tool already exists on the system.

Resolving the problem

Upgrade the Image Construction and Composition Tool instead of installing it.

Note: If you launched the installation from a mounted Windows driver, copy and run the installation locally.


6.12. Exec error 1 during installation

You receive an executable error when you try to install the Image Construction and Composition Tool on a system.

Symptoms

You receive the following error during installation:

'exec error 1'

Causes

The error indicates that port 443 is already in use and the Image Construction and Composition Tool cannot access it.

Resolving the problem

Ensure that port 443 is not in use by other applications.


6.13. Package installation error during silent upgrade

In some cases, when upgrading IBM Image Construction and Composition Tool silently, errors might occur.

Symptoms

During the silent upgrade process, the following error might be displayed:

ERROR: There is already a package installed at "/opt/IBM/icon". The installation directory must not be the same as a previously used installation directory.

Causes

This error occurs because of some mismatched product names, depending on the version that you are upgrading from and the installation methods that you used to install Image Construction and Composition Tool previously.

Resolving the problem

To resolve the problem, modify the following two lines in the file <extract_dir>/icon_silent_install_response_file.xml, by changing "Tool" to "Tools" in both cases:

<profile id='Image Construction and Composition Tool' installLocation='/opt/IBM/icon>
<offering id='com.ibm.cloud.icon.troy' profile='Image Construction and Composition Tool' 
features='icon_core' installFixes='none'/>


6.14. Incorrect synchronization parameters after Image Construction and Composition Tool upgrade

Images imported in older versions of Image Construction and Composition Tool do not have correct synchronization parameters.

Symptoms

If you imported images in a version of Image Construction and Composition Tool earlier than V1.2.0.2, they do not have the correct synchronization parameters after the upgrade.

Resolving the problem

When upgrading to Image Construction and Composition Tool V1.2.0.2, a deployment filtering function is enabled to display only the environment profiles and cloud groups that are compatible with the image to be synchronized. This filtering function is enabled only for images imported and extended after V1.2.0.2 is installed. Images that were imported or extended before the V1.2.0.2 installation do not have this function.


6.15. Universal ID and version must be unique

The combination of universal ID and version for every asset must be unique. If you extend an image and you do not enter a unique universal ID and version number combination for the newly extended virtual image, you receive an error message.

Symptoms

When you are extending a virtual image, the following error message is included in the trace.log and error.log files:

Cannot create the image: Error: HTTP response code:409
message: "CYOIG0005E: Universal ID '<ID>_<version>' is not unique."

Causes

The universal ID in combination with the version for the new virtual image that you are extending must be unique.

Resolving the problem

Enter a unique universal ID and version combination for the extended asset.


6.16. Cancel - use the current file link does not work

The Cancel - use the current file link, located on the Files to Copy section on the Install or Config tab for a software bundle, might not work correctly.

Symptoms

This problem might occur if you upload a file to the Files to Copy section on the Install or Config tab for a software bundle, save that file, and edit it. After saving and editing the file, you then select a new file to upload and click Cancel - use the current file. Although you tried to revert to the previous file, the new file is still displayed.

Resolving the problem

If the file is not being updated correctly, click Cancel . If you click OK, and see the wrong file, refresh the software bundle to omit all the changes.


6.17. VMware ESX error if you use virtual machines with snapshots

If you use virtual machines with snapshots in VMware ESX, you receive an error in the trace log file. VMware ESX cloud provider does not support virtual images where the virtual machine has snapshots.

Symptoms

You receive an error message in the trace log file that contains output similar to the following:

java.io.IOException: disk conversion of /drouter/ramdisk2/mnt/
raid-volume/raid0/templates/2/base-rhel55-64-000001-delta.vmdk 
from monolithicFlat to streamOptimized failed!
In the preceding example, the -delta.vmdk syntax indicates that there are snapshots in the virtual machine.

Causes

The VMware ESX cloud provider does not support images in which the virtual machine has snapshots.

Diagnosing the problem

You receive an error message similar to the following in the trace.log file:

[2011-10-17 17:50:52:509 EDT] 00001198 IconImageExpo 
I com.ibm.cloud.icn.execution.engine.IconImageExporter 
compressMonolithicFlatToStreamOptimizedDisk About to 
convert monolithicFlat router/ramdisk2/mnt/raid-volume
/raid0/templates/2/base-rhel55-64-000001-delta.vmdk to 
streamOptimized disk /drouter/ramdisk2/mnt/raid-volume
/raid0/tmp/exportedImages/a0e04b2c-54c6-4b84cebed0e2a37
/base-rhel55-64-000001-delta.vmdk[2011-10-17 
17:50:52:691 EDT] 00001198 IconImageExpo E com.ibm.cloud
.icn.execution.engine.IconImageExporter compressMonolith
icFlatToStreamOptimizedDisk Disk conversion failed for 
/droudisk2/mnt/raid-volume/raid0/templates/2/base-rhel55
-64-000001-delta.vmdk exited with 1[2011-10-17 
17:50:52:693 EDT] 00001198 IconImageExpo E com.ibm.cloud
.icn.execution.engine.IconImageExporter compressMonolith
icFlatToStreamOptimizedDisk disk conversion of /drouter/
ramdt/raid-volume/raid0/templates/2/base-rhel55-64-000001
-delta.vmdk failed[2011-10-17 17:50:52:693 EDT] 
00001198 IconImageExpo E com.ibm.cloud.icn.execution.eng
ine.IconImageExporter convertToStreamOptimized Unable to  convert file /drouter/ramdisk2/mnt/raid-vaid0/templates
/2/base-rhel55-64-000001-delta.vmdk from monolithicFlat 
to streamOptimized.java.io.IOException: disk conv
ersion of /drouter/ramdisk2/mnt/raid-volume/raid0/templat
es/2/base-rhel55-64-000001-delta.vmdk from monolithicFlat
 to streamOptimized failed!at com.ibm.clou
d.icn.execution.engine.IconImageExporter.compressMonolith
icFlatToStreamOptimizedDisk(IconImageExporter.java:582)>
at com.ibm.cloud.icn.execution.engine.IconImageE
xporter.convertToStreamOptimized(IconImageExporter.java:
413)at com.ibm.cloud.icn.execution.engine
.IconImageExporter.run(IconImageExporter.java:238)
at java.lang.Thread.run(Thread.java:736)

Resolving the problem

To avoid this problem, do not use virtual machines that have snapshots. Remove the snapshots from the virtual machine before using the virtual machine in IBM Image Construction and Composition Tool.


6.18. Failure during OVA disk conversion

Conversion of disk files contained in the Open Virtualization Appliance (OVA) files fails.

Symptoms

During OVA import or OVA export operations, errors occur when the disk files are converted.

Resolving the problem

Verify that the following conditions are met:


6.19. Import the same OVA image into multiple VMware ESX cloud providers fails


Symptoms

During OVA import, error message CYOES0033E is generated. For example:

CYOES0033E: Unable to import image. Another image exists with the name: 
CoC_RHEL62_64 and version: 1.0.6. If the image has already been 
imported using another cloud provider, use the Import from 
cloud provider option instead of the Import option.

Resolving the problem

Verify that the image has not already been imported for this VMware ESX cloud provider. If the image has already been imported for a different VMware ESX cloud provider, you can import the image into additional VMware ESX cloud providers. Instead of using the Import option, use the Import from cloud provider option.


6.20. Import or export of the OVA file fails


Symptoms

During OVA import or OVA export operations, errors occur. The log has the following message:

java.io.IOException: No space left on device.

Resolving the problem

Ensure that the following conditions are met:


6.21. Import a virtual machine fails

Import fails when you are importing a virtual machine.

Symptoms

During virtual machine import using the ESX cloud provider an error occurs. The log has the following message: CYOES0002E: Unable to establish a connection to #.#.#.#

Resolving the problem

Ensure that the following conditions are met:


6.22. Error occurs during OVA export


Symptoms

During OVA export operations, errors occur. You might see the following error message in the log files:

CYOCU0005

Resolving the problem

Ensure that the user specified in the user interface has write permission for the specified target folder.


6.23. OVA deployment to VMware fails with unsupported element 'Property'

If you are attempting to use VMware vSphere Client to deploy an OVA file, the operation fails.

Symptoms

When deploying an OVA template from VMware vSphere Client by using File > Deploy OVF Template, you get the following error message:

This OVF package uses features that are not supported when deploying directly to an ESX host.
The additional text of the message states:

The OVF package requires support for OVF Properties" and "Unsupported element 'Property'.

Causes

OVA deployment directly to ESX or ESXi is not supported. You must use VMware vCenter Server (VirtualCenter). To verify the server type, use VMware vSphere Client to select the server in the left column. In the right window, above the Summary tab, you check whether the server is VMware ESX/ESXi or VMware vCenter Server.

Resolving the problem

Install VMware vCenter Server (VirtualCenter) and deploy the OVA file to this VMware vCenter.


6.24. The script package maestro is in failed state

When you deploy an image that is created with IBM Image Construction and Composition Tool, you might see that the script package maestro is in a failed state.

Symptoms

After you deploy the image, the log file remote_std_err.log contains the following error message, which indicates that a prerequisite is missing:

Verifying Prerequisites - Java
ERROR: Missing IBM SDK for Java at /opt/IBM/ibm-java-i386-60

Causes

The required package is missing because IBM Image Construction and Composition Tool does not automatically install the IBM SDK for Java during image extension.

Resolving the problem

If the virtual application features provided by the maestro script package are needed, you can extend the image by using IBM Image Construction and Composition Tool and by adding a bundle for IBM SDK for Java.


6.25. Virtual image capture fails

If the virtual image capture fails, you might need to delete the failed virtual image and then create the virtual image and complete the capture process again.

Symptoms

The virtual image capture fails with the following error message:

CYOCE0002E: The capture cannot be started because the resources are 
not sufficient to capture the image. 
Check your private instance quota.  The image cannot be saved.

Resolving the problem

To enable the capture option again, expand the virtual image capture error and click the option to try to capture again. Alternatively, delete the failed virtual image and create the virtual image again. Then, complete the capture process again.

If the problem is not resolved, check the image quota of the IBM SmartCloud Enterprisecloud provider. Log on to IBM SmartCloud Enterprisevia a web browser and check how many stored images you are allowed versus how many you have. You might need to delete a stored image or increase the quota for the number of stored images.


6.26. Virtual images that contain old versions of the activation engine cannot be deployed

If you are attempting to deploy virtual images by using Image Construction and Composition Tool, the deployment of these images might stop unexpectedly.

Symptoms

When deploying virtual images in Image Construction and Composition Tool, if the activation engine version in these images is earlier than 2.1.1.24, the deployment of these images might stop unexpectedly.

Causes

If the version of the activation engine used in the image is earlier than 2.1.1.24, the file ovf-env.done might not be generated and the process cannot detect that the activation completed.

Resolving the problem

Upgrade the automation engine.


6.27. Files are not replaced correctly

If you add a file to the Files to Copy section on the Install, Config, or Rest tab of a software bundle and later replace that file, the content of the file is not updated, even though the name of the file is updated correctly in the Image Construction and Composition Tool. The content is updated correctly after you save the software bundle.

Symptoms

If you replace the file in the Files to Copy section on the Install, Config, or Resttab of a software bundle, the content of the file is not updated unless you save the software bundle. However, the name of the new file is displayed in the Image Construction and Composition Tool.

Resolving the problem

Save the software bundle before you click the file to download the contents.


6.28. The user interface stops responding while uploading files

Image Construction and Composition Tool stops responding when uploading a file that is larger than 2 GB.

Symptoms

When you create a software bundle and upload a file with a size larger than 2 GB in the Install section. You might be waiting a long time for the file to upload, while the user interface seems to stop responding.

Causes

There is a 2-GB file size limit for uploading in the browser. There is little information in the trace and error log files.

Resolving the problem

To upload large files to a software bundle:

  1. In the New file dialog box, select the Remote option and specify the URL for the file to upload in the text box. Uploads done on this dialog box are asynchronous, while local uploads are synchronous.
  2. Verify that the file is uploading by checking the status indicator for the entry in the Files to copy table.
  3. Continue to refresh the software bundle definition until the upload is completed.


6.29. Limitation when performing concurrent updates

Do not perform concurrent update operations on images, bundles, and other objects in Image Construction and Composition Tool, either using different browsers or using a browser and the command line interface.

Symptoms

The object might not be updated correctly.

Resolving the problem

Only use a single mechanism of updating an object in Image Construction and Composition Tool, either using the web interface in a single browser, or using the command-line interface.


6.30. Cannot remove software bundles from a personality

If a software bundle is added to a personality and then the virtual image is synchronized, the added software bundles cannot be removed from the personality.

Symptoms

The software bundles cannot be removed from the personality.

Resolving the problem

Delete the existing personality and create another personality where you add only the software bundles that you need.


6.31. Bundles are not displayed in the list of compatible images


Symptoms

Bundles are not displayed in the list of compatible images.

Resolving the problem

On the Add software bundles dialog box, clear the check box Show only bundles compatible with the image.


6.32. Limitation when adding bundles to a personality

For virtual images that have a single personality, all software bundles that are added to the image are automatically added to the personality. This behavior means that, in this case, the personality is identified with the image itself.

Symptoms

For virtual images that have multiple personalities, software bundles that are added to the image are not added automatically to any of the personalities in the image. For these images, you must explicitly add the software bundles to the personalities, as needed.

Resolving the problem

If you want to avoid this limitation, complete steps in the following order:

  1. Extend the virtual image with the single personality.
  2. Add software bundles, as required. The bundles are added to the personality by default.
  3. Add another personality to the image.
  4. Add more bundles to the images. The bundles are not added to any personality by default.
  5. If required, add the bundles to the appropriate personality.


6.33. Bundles not added to new images

Bundles are not added to new images in Image Construction and Composition Tool.

Symptoms

When you are creating an image from a template, bundles without an installation step defined are not added to the new image.

Resolving the problem

After you create an image from a template, add more bundles to that image


6.34. Limitation when deleting images that are synchronizing

You cannot delete an image while it is synchronizing.

Symptoms

The image is synchronizing and the synchronization is not proceeding successfully.

Resolving the problem

Do not delete an image while it is synchronizing. Wait until the synchronization process has completed and then delete it.


6.35. Time lag when deleting virtual images

If you delete a virtual image, it might take awhile for the virtual image to be completely removed from view.

Symptoms

A virtual image that you deleted is still displayed in a view in the tool. You can view and select the virtual image for some time until the deletion process has completed.

Causes

There is a refresh issue that causes a deleted virtual image to remain in view until the deletion process has completed. The deletion process can take awhile to complete.

Resolving the problem

Wait until the deletion process has completed and refresh your browser.

Note: The final time that you refresh the resource and then select the virtual image, you receive an unable to load resource error message because the virtual image has been deleted at this point. After this, the view is refreshed.


6.36. SSL exception

In the command-line interface on Linux, you receive an SSL exception error.

Symptoms

On Linux, in the command-line interface of the Image Construction and Composition Tool, you receive the SSL exception error socket.sslerror: (-1, 'SSL exception').

Causes

If you are using the command-line interface on Linux and receive an SSL exception error, there might be a compatibility issue with the Java Runtime Environment (JRE) that you are using.

Resolving the problem

Use the java -version command to verify which JRE version you are using.

To resolve the communication error, you can use a different JRE. You can download IBM JRE version 6 for Linux from the following web address: http://www.ibm.com/developerworks/java/jdk/linux/download.html.

Once you have the JRE installed, you can set environment variables to allow this new JRE to be used by the command-line interface instead of the existing JRE. For example:

export JAVA_HOME=/opt/IBM/icct/jre
export PATH=$JAVA_HOME/bin:$PATH


6.37. Cannot pass characters for arguments on the command line

You might not be able to pass certain characters for arguments on the command-line interface.

Symptoms

When invoking the command-line interface on Windows and passing arguments that contain an exclamation point (!), this character is removed from the argument during processing.

Causes

The command-line interface is based on Python. In certain cases, specific characters are used to indicate special characteristics.

Resolving the problem

When you want to pass the exclamation point as a command-line argument, use the caret character (^) to escape the exclamation point character. Also, enclose the argument in double quotation marks. For example, instead of "password!", use "password^!"


6.38. TCP/UDP firewall ports might be appended incorrectly on Red Hat system after synchronization

The TCP/UDP firewall ports might not be appended correctly on a Red Hat Linux system after synchronization.

Symptoms

If a software bundle has firewall rules defined when the bundle was added to a Red Hat image through the VMware ESX cloud provider, the TCP/UDP firewall ports might be appended incorrectly after synchronization.

Resolving the problem

Perform the following steps:

  1. Run: service iptables save
  2. Run: service iptables stop
  3. Run: cd /etc/sysconfig. Open the iptables file. If the new firewall rules are appended after the reject rules, move the new firewall rules before reject rules. If the new firewall rules are UDP port instead of TCP port, change tcp to udp for the new firewall rule.
  4. Run: service iptables start


6.39. Virtual image synchronization problems

Determine the reason why the virtual image synchronization fails.


6.39.1. Virtual image synchronization takes a long time

Your virtual image synchronization process takes a long time to complete.

Symptoms

Virtual image synchronization takes a long time and might time out and fail.

Causes

The cloud performance is poor. Cloud performance is not always consistent and can vary depending on usage and your connection. Depending on the cloud performance at any particular time, the virtual image synchronization can take longer to complete. Image Construction and Composition Tool continues the synchronization process in case there are performance issues. If the synchronization process is very slow and the performance is particularly poor, the synchronization eventually times out and you are notified that the synchronization failed because of performance issues.

Resolving the problem

After the virtual image synchronization process has timed out, you can retry the synchronization. If you retry and are still having synchronization issues, check the log files to determine the particular cloud provider issue that is affecting synchronization.


6.39.2. Virtual image synchronization fails after five hours for images

Virtual image synchronization fails after five hours for images on the IBM PureApplication System and IBM SmartCloud Provisioning cloud provider.

Symptoms

Virtual image synchronization fails after five hours even if the image deployment is still running on IBM PureApplication System.

Causes

By default, the Image Construction and Composition Tool waits up to five hours for the deployment of an image on IBM PureApplication System or IBM SmartCloud Provisioning. If the deployment does not complete within that time, the Image Construction and Composition Tool returns a failure.

Resolving the problem

In the configuration.config file in the <ICCT_INST_DIR>/icn.app/config/ directory, edit the parameter maxIWDDeployWaitTime to increase the wait time. The default value is 300 minutes which is five hours.

Restart the Image Construction and Composition Tool to apply the changes.


6.39.3. Synchronization fails for a virtual image imported from a running virtual machine

If you import a virtual image from a running virtual machine, the virtual image does not synchronize if Network Manager is running on the virtual machine. Network Manager must be disabled on the virtual machine.

Symptoms

The virtual image does not synchronize.

Causes

Network Manager has not been disabled on the virtual machine.

Resolving the problem

Disable Network Manager on the virtual machine and import the virtual image again.

To disable network manager, complete a procedure similar to the following:

  1. From the command line, run the following:

    service NetworkManager stop 
  2. Run the following command:

    chkconfig NetworkManager off
    
  3. Ensure that the interfaces are not controlled by Network Manager by editing the following items in the /etc/sysconfig/network-scripts/ifcfg-eth0 file:

    NM_controlled = no  
    and
    ONBOOT=yes
    

After you have disabled Network Manager on the virtual machine, try to import and synchronize the virtual image again.

There are many reasons for a synchronization failure. To help determine the problem, check the Image Construction and Composition Tool log files to debug.


6.39.4. Virtual image synchronization when using a trial version of VMware ESX

If you attempt to synchronize an image using a trial version of VMware ESX, the synchronization fails when attempting to copy the disk images to the VMware data store.

Symptoms

Image synchronization fails and the log contains messages similar to the following example:

Caused by: (0)null
at org.apache.axis.transport.http.HTTPSender.readFromSocket(HTTPSender.java:744)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)
or

java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: 
com.ibm.venture.vm.VirtualizationShimException: Remote exception. No details available. 
shim=https://localhost/sdk_copy26 endpoint=https://localhost/sdk_copy26 
localizedMessage="Remote exception. No details available." 
causeType=org.apache.axis.AxisFault cause=(0)null

Causes

The VMware ESX server is a trial version. Only licensed VMware ESX servers are supported.

Resolving the problem

Use a licensed version of VMware ESX for your VMware ESX cloud provider.


6.39.5. Virtual image synchronization fails because of return code value

Virtual image synchronization fails if you use a script for the bundle installation that produces a return code value that is greater than zero. The Image Construction and Composition Tool checks the return value of the installation scripts used in bundles. A zero return code indicates success and the synchronization process can proceed.

Symptoms

Virtual image synchronization fails with the error: Unable to initiate add of execution bundle to VM.

Causes

The script used to install software in a bundle does not return a zero value.

Resolving the problem

Make sure that the scripts that you are using to install the software return a zero value to indicate successful installation. A value greater than zero indicates a failure and the virtual image cannot be synchronized. The installation script checks for error conditions relevant to the installation of the bundle. For errors that might compromise the bundle installation, an exit code greater than zero must be returned, indicating failure. If a script returns no exit code explicitly, the exit code of the last command of the script is implicitly returned. Make sure that the bundle installation script can handle parameters passed in the following format:

scriptname -parameter1 <parameter1 value> -parameter2 <parameter2 value>
A sample installation script is as follows:

#!/bin/sh
#
# Licensed Materials - Property of IBM # (C) Copyright IBM Corp.  2011
# All Rights Reserved
# US Government Users Restricted Rights -Use, duplication or 
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
# ----------------------------------------------------------------
#
# Description:
#   Sample install script for an ICON bundle
#
#
 
function usage () 
{
	echo ""
	echo " Usage :" 
	echo "     installBundleSample.sh -parm1 \"parm1value\" -parm2 \"parm2value\"" 
	echo ""
	exit 1
}
 
echo "$0: === Starting Install of Bundle Sample === "
echo
 
echo "$0:    Arguments are: $* "
echo
 
#Init
MY_PARM1="Default parm1 value"
MY_PARM2="Default parm2 value"
 
echo "$0:    Parsing Arguments "
echo
#Read arguments
while [ $# -ne 0 ]
do
    case $1 in     -parm1*)
       MY_PARM1=$2
       ;;
    -parm2*)
       MY_PARM2=$2
       ;;
    *)
       ;;
esac
shift 1
done
 
# Now perform the work and actually do the install stuff - check the return code echo "$0:    About to do the real work..."
echo parm1 is "$MY_PARM1"      parm2 is "$MY_PARM2"
if [ $? -gt 0 ]; then    echo "$0: echo $MY_PARM1 $MY_PARM2 failed" >&2
   exit 1
fi
 
 
 
# finish up and return zero to indicate everything went ok 
#     (non-zero return code signifies an error)
echo " "
echo "$0: Done."
exit 0


6.39.6. Virtual image synchronization fails and troubleshooting is not possible

If the synchronization fails, it is difficult to troubleshoot the problem because the deployed image and the bundles used for synchronization are removed.

Symptoms

The synchronization fails and it is difficult to perform problem determination because the deployed image and the compressed package are removed. Problems with the bundle installation scripts are difficult to troubleshoot.

Resolving the problem

Use the ICCT_DEBUG variable to troubleshoot the problem.

If this variable is not defined on your system, you can set it up by following these steps:

  1. If Image Construction and Composition Tool is started, stop it:

    cd <install_dir>
    ./stop.sh
    
  2. Define the ICCT_DEBUG variable:

    export ICCT_DEBUG=true
    
  3. Restart Image Construction and Composition Tool:

    ./start.sh
    

If later you no longer need the ICCT_DEBUG variable, you can disable it:

cd <install_dir>
./stop.sh
export ICCT_DEBUG=
./start.sh


6.39.7. Windows image synchronization fails

Synchronization of a Windows image fails.

Symptoms

Windows image synchronization to the virtual machine can fail if the /etc/hosts file, required by the RXA protocol, in the Image Construction and Composition Tool server configuration is not configured correctly.

Resolving the problem

In addition to the localhost entry, the machine where the Image Construction and Composition Tool is installed must contain an entry with the Image Construction and Composition Tool server IP and the related host name and fully qualified host name, for example:

9.168.12.202  nc12202        nc12202.it.ibm.com


6.39.8. Cannot synchronize Windows 2008 R2 image

If an image is extended from another image, which is imported from a Windows 2008 R2 virtual machine running, the image synchronization might fail.

Symptoms

The synchronization fails and the image status is Virtural machine failed to start. If you open the VMware vCenter console, a window is displayed with the error

autochk program not found - skipping AUTOCHECK
If the synchronization fails, Image Construction and Composition Tool rolls back the changes, and the deployed virtual machine becomes Orphaned as a part of the rollback. The error window disappears.

Resolving the problem

This problem is resolved in VMware ESX 4.1 Update 2 and VMware ESX 5.0 Update 1. For more information, see the following article in the VMware Knowledge Base: Cloned Windows 2008 R2 virtual machine fails to boot with the error: autochk program not found - skipping AUTOCHECK.


6.39.9. Timeout with Linux bundle in Windows image on VMware

Timeout with no error when synchronizing a Linux bundle inside a Windows image on a VMware cloud provider

Symptoms

Synchronizing a Linux bundle inside a Windows image on a VMware cloud provider times out with no error.

Causes

If the installation script used in a software bundle is not recognized by Windows as a known executable, the installation process executed during synchronization runs without stopping. The synchronization task fails after the timeout.

Resolving the problem

The timeout expires after six hours and the synchronization fails. At this point, you can remove the incorrect Linux bundle and add the correct one.


6.39.10. Image synchronization from the command line does not work

If you attempt to use the command line (CLI) to synchronize an image, the operation fails.

Symptoms

When you are using the CLI to synchronize a virtual image with an IBM PureApplication System or IBM SmartCloud Provisioning cloud provider, the image synchronization fails.

Resolving the problem

  1. To get a list of all possible parameters, run the following command:

    allCloudSyncParams = image.getSynchronizeParametersFromCloudProvider()
    
    This command returns a list similar to the following example:

    [
      {
        "label": "Select an environment profile for deployment:",
        "validValues": [
          {
            "label": "envProfile10 (PureSystems_ESX)",
            "validValues": [
              {
                "label": "CloudGroupLarge",
                "validValues": [
                  {
                    "label": "ipgroup10",
                    "value": "1013"
                  }
              ],
              "isPassword": "False",
              "description": "cloud group",
              "type": "string",
              "value": "12",
              "name": "ipGroup"
            }
          ],
          "isPassword": "False",
          "description": "environment profile",
          "type": "string",
          "value": "10",
          "name": "cloudGroup"
        },
        {
          "label": "none",
          "validValues": [
            {
              "label": "CloudGroupLarge (PureSystems_ESX)",
              "validValues": [],
              "isPassword": "False",
              "description": "cloud group",
              "type": "string",
              "value": "12",
              "name": "ipGroup"
            }
          ],
          "isPassword": "False",
          "description": "environment profile",
          "type": "string",
          "value": "0",
          "name": "cloudGroup"
        }
      ],
      "isPassword": "False",
      "description": "select an environment profile.",
      "type": "profile",
      "name": "profileList"
      }
    ]
    
  2. Based on the retrieved list, build a new JSON object with a structure similar to the following example:

    myCloudSyncParams =
      [
        {'name':'profileList', 'type':'string', 'value':''},
        {'name':'envprofile', 'type':'string', 'value':'10'},
        {'name':'cloudgroup', 'type':'string', 'value':'12'},
        {'name':'ipgroup', 'type':'string', 'value':'1013'},
        {'name':'username', 'type':'string', 'value':'root'},
        {'name':'password', 'type':'string', 'value':'passw0rd'}
      ]
    
    In this JSON object:

    • The profileList element has an empty value.
    • You must add the username and password elements to the JSON object. They are not returned by the getSynchronizeParametersFromCloudProvider() method.

    For assistance in building this JSON object, use the command line with the -f parameter and sample file sample/createSynchronizeParametersForCloudProvider.py.

  3. To get the list of possible image parameters, run the following command:

    myImgSyncParams = getSynchronizeParametersFromImage()
    
  4. To synchronize the image, run the following command:

    image.synchronize(myImgSyncParams, myCloudSyncParams, True)
    

For assistance in building this JSON object, use the command line with the -f parameter and sample file sample/createSynchronizeParametersForCloudProvider.py.


Troubleshoot Advanced Middleware Configuration

Discover solutions or workarounds for common problems.


7.1. Troubleshoot the generation process

If the generation process does not create the required objects on the framework server, try these common solutions.

Make sure you have met all the prerequisites

Make sure generation completed successfully

  1. Check the output of the generation process for error messages.
  2. Open the RAFW_HOME/user/preserve/envgen/env-config.properties file and make sure you can read it.

Check the services log

  • If the use Tomcat as your application server, the default log directory is /opt/IBM/RAFServer/server/tomcat/logs.
  • If you use WebSphere Application Server, the information is sent to the SystemOut.log file in the logs directory of your server.

Verify your permissions

  1. Make sure the application server user (either the Tomcat user or the WebSphere Application Server user) has write access to the work, user, and log directories.
  2. Make sure you can SSH into the target system using the target operating system user (OS_USER) credentials from the configuration definition.
  3. Make sure the target operating system user has permission to write to the OS_RAFW_HOME directory on the target system.


7.2. Clearing Jython cache

If the Jython cache files on a remote system are out of date, clear the Jython cache by running the rafw_delete_jython_class_files action at the node scope of the remote system. The error message that indicates that the Jython cache is out of date is shown below. In particular, the ImportError and NameError lines in the error message indicate the problem.

call_wsadmin_function:
     [exec] WASX7209I: Connected to process "dmgr" on node dmgr using SOAP connector;  The type of process is: DeploymentManager
     [exec] WASX7017E: Exception received while running file "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/jythonLib.py"; exception information: com.ibm.bsf.BSFException: exception from Jython:
     [exec] Traceback (innermost last):
     [exec]   File "<string>", line 71, in ?
     [exec]   File "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/BFWCommand.py", line 18, in ?
     [exec]   File "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/ConfigMediator.py", line 17, in ?
     [exec]   File "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/ConfigComparor.py", line 16, in ?
     [exec] ImportError: No module named bfw
     [exec] 
     [exec] WASX7015E: Exception running command: "BFWCommand.execute('-scope cell -scopename qa -nodename dmgr -properties /opt/IBM/RAFServer/rafw/user/environments/cycle/cells/qa/jms.xml -mode import -marker JMSProviders', ['JMSProvider'])"; exception information:
     [exec]  com.ibm.bsf.BSFException: exception from Jython:
     [exec] Traceback (innermost last):
     [exec]   File "<input>", line 1, in ?
     [exec] NameError: BFWCommand
     [exec] 


Procedure

At the node scope of the target system, run the rafw_delete_jython_class_files action by entering the following command: ./rafw.sh -e env_name -c cell_name -n node_name -t rafw_delete_jython_class_files.


7.3. Test server connections

To verify that the network setup is completed correctly, test the connection between the framework server and the target computers. The framework server and the target computers can be UNIX, Linux, or Windows systems.

Target systems on UNIX communicate with the framework server by using the SSH protocol with either username and password or SSH key authentication. Keys can be managed using the standard operating system utilities. your framework server is a Windows system and the target computer is a UNIX or Linux system, you must use third party software to test SSH connections.


Procedure

To test the framework server and target system connection:


7.4. Connection to UNIX or Linux systems fails with 'Broken pipe' message

If interactive commands exist in the user profile, a secure shell (SSH) connection to remote UNIX or Linux systems fails with a broken pipe message in the Remote Execution and Access (RXA) log file. To troubleshoot this error, you must remove the interactive commands from the user shell profile.


Diagnosing the error

If connection to a remote UNIX or Linux system fails, determine if the error is due to interactive commands in the user profile by performing the following steps:

  1. Enable RXA tracing.
  2. From the framework server, run an action that requires a connection to the target UNIX or Linux system.
  3. In the RXA trace output, look for the following lines: java.io.IOException: Read end dead and Broken pipe.
  4. On the framework server, enter the following command: ssh remoteuser@remote.server.name uname. If prompted, enter the password.

    • If no interactive commands exist in the user profile, the command returns only the output of the uname command, for example, Linux.
    • If interactive commands exist in the user profile, the command returns additional data in the output, such as stty: statndard input: Invalid argument.


Troubleshoot the error

If the connection error is due to interactive commands in the user profile, remove the interactive commands from the user profile on the remote system. Based on the shell specified for the remote user, the files to edit might be different. For example, for users of the bash shell, the files to edit might be /etc/profile ~/.bash_profile and /etc/profile ~/.bashrc.

Verify that the interactive commands are removed by entering the command in step 4 above. The command output must be the value of uname only.


7.5. Troubleshoot installation errors

If your installation actions fail or return errors, you might want to try some common troubleshooting techniques.

Review the installation prerequisites

Ensure that you meet the installation prerequisites, such as the software and hardware requirements, to install the product.

Review the installation response files

If your installation actions fail, locate and read the installation response file for the product. The installation response file supplied by the product contains information about the parameters required for the installation. Installation failure or errors might be due to missing or incorrectly-supplied installation parameters.


7.6. Troubleshoot Workload Deployer errors

Troubleshoot errors with PureApplication System processes by referring to the described problems and the possible resolutions. If the problem cannot be resolved, you must restart the process.

List of errors

Process Problem Resolution
Connect to server Script Package log shows the error: "Could not send Message."

  1. Verify the host, domain, username, password, and SSL options.
  2. Verify that the Advanced Middleware Configuration for PureApplication System server is available by logging in to the web client.
  3. Retry the pattern deployment.

Create a new Advanced Middleware Configuration configuration environment Configuration environment and projects not created Check the logs for details:

  1. Check the services log at /opt/IBM/RAFServer/server/tomcat/logs/raf.2012-03-16.log
  2. Check the server log at /opt/IBM/RAFServer/server/tomcat/logs/catalina.out
  3. Check the framework server log at /opt/IBM/RAFServer/server/tomcat/logs/framework.log.0

  • Create a new Advanced Middleware Configuration configuration environment
  • Update the Advanced Middleware Configuration environment pattern to match a newly supplied pattern.

Advanced Middleware Configuration server is unable to connect to the newly deployed target system. Check product logs and enable tracing:

  1. Check the framework server log at RAFW_HOME/logs/framework.log.0
  2. Check the action log at RAFW_HOME/logs/scope/action_name.trace
  3. Enable RXA tracing and try connecting to the server again.

Fix errors, if any, in the job:

  1. Check job results in the Advanced Middleware Configuration web client and fix any errors.
  2. Restart the job and verify that it completes successfully.
  3. Retry the pattern deployment.

Permission denied to create working directory on the target host.

  1. Edit the value for RAFW_HOME_PATH in the script package settings to match the value for LINUX_RAFW_HOME in the file RAFW_HOME/configure.properties.
  2. Retry the pattern deployment.

Find the automation plan name for a Workload Deployer cell The Advanced Middleware Configuration environment was not created (see previous row).

  1. Verify that the RAFW_env_cell_execute or RAFW_env_cell_import automation plan exists on the Advanced Middleware Configuration server
  2. Verify that the user defined as the RAFW_USER has access to view the environment group.
  3. Retry the pattern deployment.

Launch the project Errors when the project runs.

  1. Check job results in the web client and fix any errors.
  2. Restart the job and verify that it completes successfully.

You do not need to retry the pattern deployment because this is the last step in the script package.


7.7. Missing configuration in Advanced Middleware Configuration console after upgrade

After you upgrade to Advanced Middleware Configuration version 1.1, an existing configuration definition might not be visible in the Advanced Middleware Configuration console.

The configuration properties file is named env-config.properties. In versions of Advanced Middleware Configuration previous to 1.1, the configuration properties file was in the RAFW_HOME/work directory. Beginning with version 1.1, the configuration properties file is in the RAFW_HOME/user/preserve/envgen/ directory. The console searches the envgen directory when displaying available configurations on the Configuration page. Configuration properties files created in the work directory might not be displayed.

  1. If you can see the missing file in theRAFW_HOME/work directory, copy the file to the RAFW_HOME/user/preserve/envgen/ directory.

  2. If you cannot see the missing file in the RAFW_HOME/work directory, but the configuration definition exists in the RAFW_HOME/user/environments directory, you can recreate the configuration properties file. Run the existing configuration option of the configuration editor and point it to the live configuration that existing configuration definition represents. You do not need to generate the configuration definition after you finish editing the configuration properties file.


7.8. Automation plan fails with no output in results

If your automation plan fails, but the Results display contains no messages, there might be a problem with the RAFW_LOCAL server for the automation engine.

  1. Start the automation web client by opening a browser and specifying the host name for your framework server followed by a slash (/) and the letters "bfui", as in the following example:

    framework.server.example.com/bfui
    
  2. On the Console page, go to the menu on the left side of the page and click Servers. The RAFW_LOCAL server should be displayed with a green icon to indicate that it is running.
  3. If the server is not running, click the server name. The server information is displayed in the lower pane.
  4. Click Test Connection. A server test is run and the results are displayed on the Test Results page.
  5. If you see a Status: Authentication failure message, complete the following steps:

    1. Click Server Auth on the left side of the page.
    2. Click RAFW_LOCAL to display the server authorization information for the server.
    3. Enter the correct password for the displayed user in response to the Password and Verified prompts.
    4. Click Servers again.

    This problem is commonly caused by someone changing the Server Auth user password, but not updating the password in the web client.

  6. If you see a Status: Connection failure message, complete the following steps:

    1. Ensure that the Host name or IP address shown on the Details page is correct.
    2. Ping the server and make sure it is responding.
    3. Verify that the automation agent is running:

      • On Linux, enter the following command:

        ps -ef | grep bfagent
        
      • On Windows, check the Task Manager for the bfagent task.

      If the agent is not running, browse to the Rational Build Forge Information Center and search for "Troubleshooting agents."


7.9. Garbled data in .properties file

Java .properties files are encoded by default as ISO-8859-1. Advanced Middleware Configuration requires that .properties files be in UTF-8 encoding.

Advanced Middleware Configuration recognizes standard .properties files, such as the following files, at any scope, and overrides the default Java encoding:

The problem occurs when you create a new .properties file with a name that is not recognized. For example, you might copy a promote.properties file to a template and rename the file my_data.properties.

The solution is to manually change the file encoding by completing the following steps:

  1. In the Eclipse client, right-click the file name and click Properties.
  2. Under Text file encoding, click Other and then select UTF-8.
  3. Click OK to close the window.
  4. You might see a warning message saying the new encoding conflicts with the content type. Click Yes to continue with the change.


8.1. Error messages overview

Most of the Rational error messages begin with a code such as CRRE and end with a message type such as E or I. Use the prefix to isolate the probable cause or location of the error and the message type to determine the severity of the error. Any other kind of error message relates to the databases, application servers, and other components used by the Rational product. Sometimes, the messages have an RSS feed that you can subscribe to. This topic explains the anatomy of the error messages.


Error message prefixes

Use the message prefix to identify the component that caused the error. These prefixes are specific to Rational Automation Framework:

CRWBP

IBM WebSphere Enterprise Service Bus, IBM WebSphere Process Server , IBM WebSphere Service Registry and Repository

CRWEB

Automation web client

CRWFW

Automation framework

CRWIH

IBM HTTP Server

CRWSC

Automation services client

CRWSE

Automation services server

CRWVE

IBM WebSphere Virtual Enterprise

CRWWA

IBM WebSphere Application Server

CRWWP

IBM WebSphere Portal


Error message suffixes

Use the message suffix to determine the severity of the error. These suffixes are used in the error messages:

A

Audit message, indicating the results of an audit.

C

Configuration message, indicating the current product configuration. This message does not indicate configuration being managed by the product in other systems, but instead displays the configuration of the product.

E

Error message, alerting you to a problem that must be solved before you can continue with your task.

I

Informational message, describing status or providing information for normal conditions and operations.

W

Warning message, alerting you to a condition that might cause problems in the future. You can generally continue with your tasks, but those tasks might not complete in a way that is expected.

Sometimes, even though the error message has a code similar to CRW, the error might be caused by a database, application server, or another product used with Rational Automation Framework. To accurately isolate the cause of the error, see the following log files:

For specific log file locations, check the product documentation for the appropriate product. For example, you can find WebSphere Application Server logs in the WAS_PROFILE_HOME directory.


RSS feed

If the message has an RSS feed and if you subscribe to the feed, you are notified every time that a technical document relating to the error goes live on World Wide Web. For more information about RSS feeds, visit the "Learn more about RSS feeds" web page at http://www.ibm.com/ibm/syndication/us/en/howto.html.