Troubleshooting
- Enable account access for the IBM service representative
- Obtaining a collection set of problem determination information for IBM support
- Create a shell account
- Use diagnostic tools
- View and download system log files
- Collecting log files for hypervisor image troubleshooting
- Diagnosing Advanced Middleware Configuration problems with message logs
- Diagnosing Advanced Middleware Configuration problems with message logs
- Enable Advanced Middleware Configuration tracing
- Advanced Middleware Configuration logs
- Rules for the standard output stream
- Location and description of product logs
- View product logs
- Log customization
- Log configuration for the server-side component of the Eclipse client
- Log level settings
- Global logging component
- Customize logging properties for action trace logs
- Customize logging properties for framework server trace logs
- Customize logging for Jython classes
- Customize logging properties for the Tomcat server
- Monitor LED status
- Monitor system events
- Monitor system problems
- Manage trace settings
- Configure tracing for the workload agent
- Troubleshoot by symptom
- CMM event occurs when compute node temperatures differ
- Troubleshoot LDAP connection issues
- Hypervisor errors and scenarios that can cause them
- Self-describing capability does not work for an external PureApplication System Agent
- Troubleshoot the System Monitoring service
- KFWITM088W Additional Topology information not available
- No monitoring data for the PureApplication System
- TEPS connectivity error
- System Monitoring service remains in launching state for over an hour
- Error occurs when middleware monitoring shared service is deployed without a running System Monitoring shared service
- Refresh a WebSphere Application Server configuration with the proxy shared service
- Environment profile not available for virtual application pattern deployment
- Virtual systems deploy successfully with failed scripts
- Situation events not forwarded to event receivers
- Troubleshoot Image Construction and Composition Tool issues
- Log files for troubleshooting
- Log files for synchronization
- Adjusting the configuration timeout settings
- Set logging levels for troubleshooting
- VMware ESX cloud provider troubleshooting
- Troubleshoot problems during VM activation
- VMware ESX cloud provider creation fails
- Work with enablement bundles
- Update Activation Engine in a virtual image
- Image Construction and Composition Tool does not respond
- Image Construction and Composition Tool does not start
- Image Construction and Composition Tool returns EOFException
- Install executable launcher error
- Install executable error during installation
- Package installation error during silent upgrade
- Incorrect synchronization parameters after upgrade
- Universal ID and version must be unique
- Cancel - use the current file link does not work
- VMware ESX error if you use virtual machines with snapshots
- Failure during OVA disk conversion
- Import the same OVA image into multiple VMware ESX cloud providers fails
- Import or export of the OVA file fails
- Import a virtual machine fails
- Error occurs during OVA export
- OVA deployment to VMware fails with unsupported element 'Property'
- The script package maestro is in failed state
- Virtual image capture fails
- Virtual images cannot be deployed
- Files are not replaced correctly
- The user interface stops responding while uploading files
- Limitation when performing concurrent updates
- Cannot remove bundles from personality
- Bundles are not displayed
- Limitation when adding bundles to a personality
- Bundles not added to new images
- Limitation when deleting images that are synchronizing
- Time lag when deleting virtual images
- SSL exception
- Cannot pass characters for arguments
- TCP/UDP firewall ports might be appended incorrectly on Red Hat system after synchronization
- Virtual image synchronization
- Virtual image synchronization takes a long time
- Virtual image synchronization fails after five hours for images
- Synchronization fails for a virtual image imported from a running virtual machine
- Virtual image synchronization when using a trial version of VMware ESX
- Virtual image synchronization fails because of return code value
- Virtual image synchronization fails and troubleshooting is not possible
- Windows image synchronization fails
- Cannot synchronize Windows 2008 R2 image
- Timeout with Linux bundle in Windows image on VMware
- Image synchronization from the command line does not work
- Troubleshoot Advanced Middleware Configuration
7.1. Troubleshoot the generation process
7.2. Clearing Jython cache
7.3. Test server connections
7.4. Connection to UNIX or Linux systems fails with 'Broken pipe' message
7.5. Troubleshoot installation errors
7.6. Troubleshoot Workload Deployer errors
7.7. Missing configuration in console after upgrade
7.8. Automation plan fails with no output in results
7.9. Garbled data in .properties file- Rational Automation Framework error messages
Troubleshoot and support
Use troubleshooting information to isolate and determine issues that you might be encountering.
You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps. The steps to troubleshoot an issue vary depending on the problem. To help make relevant information available to you as quickly as possible, the log files and other tools for troubleshooting problems are consolidated. Not all steps are relevant for every problem.
To call IBM Support to resolve an issue, remember to specify the machine type model (MTM) number and the serial number (SN) of the system, rather than those of an individual component. These details are used to connect you with the appropriate IBM service representative who has PureApplication System expertise.
A note about Storehouse Browser: If an issue requires contacting IBM Support, the representative might ask you to locate component or other resource information that is stored in the PureApplication System Storehouse. To find that information, use the Storehouse Browser, which you can access with the system console.
- Locate the MTM and SN to provide to the IBM Support. These numbers are displayed together on the top-right corner of the physical system in the following format:
<MTM_digit> - <SN_digit>, where X is an MTM digit, and Y is an SN digit.MTM reference: See the following table to confirm the correct MTM of the system. As column 2 of the table indicates, the only configuration option that differentiates the offerings is the number of compute nodes.
SNs for MTM 8283 12X-3S2
MTM-SN Number of compute nodes Location of compute nodes 8283-12X 2 Chassis 1: Bays 5 and 6 8283-14X 4 Chassis 1: Bays 5, 6, 7, and 8 8283-3C2 6
- Chassis 1: Bays 3 and 4
- Chassis 2: Bays 3 and 4
- Chassis 3: Bays 1 and 2
8283-3F2 12
- Chassis 1: Bays 3, 4, 5 and 6
- Chassis 2: Bays 3, 4, 5 and 6
- Chassis 3: Bays 1, 2, 3 and 4
8283-3L2 24
- Chassis 1: Bays 3 - 10
- Chassis 2: Bays 3 - 10
- Chassis 3: Bays 1 - 8
8283-3S2 38
- Chassis 1: Bays 3 - 14
- Chassis 2: Bays 3 - 14
- Chassis 3: Bays 1 - 14
- Contact IBM.
- In the U.S. and Canada, call 1-800-IBM-SERV (1-800-426-7378). Select the second VRU option for software.
- To locate a global IBM support contact, see the Directory of worldwide contacts at http://www.ibm.com/planetwide/. Select the contact number or VRU option for software.
Enable account access for the IBM service representative
Use the system console to enable access for the IBM service representative to service the system.
You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps.
After a Problem Management Record (PMR) is opened, you are asked to prepare the system for service by an IBM service representative. Use the system console to enable the IBM service representative account, which is used by the IBM service representative to log in to the system console for the purpose of servicing the system. The account remains enabled for 36 hours. It becomes disabled after 36 hours has passed.
Procedure
- Access the console pane.
Click System Console > System > Troubleshooting.- Select the Enable IBM service representative account access check box.
- Click Generate to create a secret key. The encrypted symmetric key populates the Secret key field. You must provide this key to IBM to request the account credentials. This key is used by the IBM Service Center for a time-limited password.
- Contact IBM. Specify the machine type model (MTM) and the serial number (SN) of the system, rather than those of an individual component.
- In the U.S. and Canada, call 1-800-IBM-SERV (1-800-426-7378). Select the second VRU option for software.
- To locate a global IBM support contact, see the Directory of worldwide contacts at http://www.ibm.com/planetwide/. Select the contact number or VRU option for software.
Obtaining a collection set of problem determination information for IBM support
Use the console to obtain a set of problem determination information to assist with debugging problems with the system. IBM support provides you with an IP address to use when completing the steps in this topic.
You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps.
Procedure
- Export system events:
- Click System Console > System > Events.
- Click the Export icon and follow the prompts to save the event information.
- Create an image of the infrastructure map:
- Click System Console > Hardware > Infrastructure Map.
- Save a screen capture of the hardware map.
- Obtain the IP collection set. This collection set contains trace and log files that IBM support uses to help diagnose and debug any problems. These size of these files is large, and you must use FTP to provide them to IBM support.
- Click System Console > System > Troubleshooting.
- Expand the System Log section, and click Collect Log.
- The default collection set is Management. Change this setting to IP. Enter the IP address that has been provided by IBM support. Click OK.
- When the state of the request changes to Available click the Download icon, and follow the prompts to save the file.
- Contact IBM Support to open a Problem Management Record (PMR). For information on contacting IBM support, refer to the instructions provided in Contacting IBM support.
- Use the PMR number that IBM support provides, send the problem determination documentation that you have gathered. Follow the steps that are outlined in Exchanging information with IBM PureApplication System technical support.
Note: Do not send any confidential information from your organization.
Create a shell account
A shell account must be used to access the system's troubleshooting features. After you create the shell account, use the help command to retrieve a list of available troubleshooting commands.
You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps.
Procedure
- Use the console to create the shell account. Click System Console > System > Troubleshooting.
- Expand Shell account.
- Enter a password in the Password of current user field and click Create.
4.1. View and download system log files
The system contains stored log data that you can view directly from the system console, or you can download the information to your local file system.
You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps.
You can view and download two log files: error.log and trace.log. Each log file has a maximum size that it cannot exceed and a file limit defining the number of versions of that file that can be maintained on the system. Each error.log file can be a maximum of 2 MB in size and up to five versions of the file are maintained for a total of 10 MB of available data. Each trace.log file can be a maximum size of 100 MB in size and up to 20 versions of the file are maintained (1 active file and 19 archived files). If a version of either log file is older than 30 days, it is automatically deleted.
Procedure
- Complete the following steps to view and download logs from the workload console:
- Click System > Troubleshooting.
- Choose from the following options:
- Kernel Service log file
- View the log files for kernel services. Kernel services trace activity related to virtual application deployments, not virtual systems.
- Storehouse log file
- View the log file for the Storehouse Browser. The Storehouse Browser is a repository used as a key component for troubleshooting problems. The Storehouse log file contains activity related to Storehouse database access used for virtual application deployments.
- View current error file
- View details about the current list of high level messages (ERROR, WARNING, INFO) from the main component which manages images, patterns, the user interface, and all stages of deployments. By choosing this option, a new browser view opens and streams all new content written to the error file in real time until you close or pause the view.
- View current trace file
- View details about the current trace file. In addition to containing the same content as the error file, the trace file contains all enabled trace levels (FINE, FINER, FINEST, ALL) by component. By choosing this option, a new browser view opens and streams all new content written to the trace file in real time until you close or pause the view.
- Download log files
- Download a trace.zip file that contains all log files, both current and archived.
When the file size limit is reached, the file is compressed and archived to a *.log.gz file, for example:
-rw-r--r-- 1 root root 7086206 Jan 11 11:25 trace_2013.01.11_11.25.09.0707.log.gz -rw-r--r-- 1 root root 7048265 Jan 11 13:22 trace_2013.01.11_13.22.44.0406.log.gz -rw-r--r-- 1 root root 59042938 Jan 11 14:23 trace.logThen, data is logged to a new, uncompressed file.- Download the latest log files only
- Download a trace.zip file that contains only the current, uncompressed log files and not the archived files. By choosing this option, you can download a significantly smaller trace.zip file in less amount of time than when you download a trace.zip file that contains both current and archived files.
- Remove old traces and logs
- Delete the archived files for workload console logs only. The currently active files remain available.
- Complete the following steps to download a collection of logs from the system console:
- Click System > Troubleshooting.
- Expand the System Logs section, and click Collect System Logs.
- From the Request System Logs panel, select one of the following collection sets, and click OK:
- System
- Collects logs for the system including:
- Management collection logs
- Deploy collection logs
- System configuration (status and inventory).
- Management
- Collects PureApplication System management logs. This collection contains data to help resolve problems with the system console and runtime.
- Deploy
- Collects workload console logs. This collection contains data to help resolve problems with workload management. For diagnostics on any virtual images, use the viewer within the workload console.
- Complete
- Collects the following:
- System collection set
- Dumps collection set
- Logs from the management node
- Dumps
- Collects JVM diagnostic information from the PureSystems. Manager. This collection set contains diagnostic information from the JVM which supports the system console functions. This information is primarily used by the IBM support team.
- IPaddress
- Select the IP collection set to use from the provided list. Diagnostic data is collected from specific components including Chassis Management Module(CMM), IBM Storwize V7000, IBM RackSwitch. G8264, IBM Flex System EN4093 10Gb Ethernet Scalable Switch, IBM Flex System FC5022 16Gb SAN Scalable Switch, management nodes and compute nodes. Complete the following steps to obtain the relevant IP address for the component for which logs are required:
- Use the following REST API URI to return a list of rack components: http://<IP address>/admin/resources/systemlogs?ipmap
- Examine the REST response to identify the relevant IP address for the component for which you require logs. Note that all IP addresses that are listed for one particular component are valid.
For assistance in determining the IP address, contact IBM Support.
The data is generated in a table format, which includes the following actions that you can choose from:
- Export
- Export a systemlogs.csv file to your local system. The file contains a summary of the log collections already created.
- Refresh
- Regenerates values contained in the table.
- Download
- Download the log collection set to your local system. The generated logs are in compressed archive format (tgz).
- Delete
- Delete the log collection set.
4.2. Collecting log files for hypervisor image troubleshooting
When you create a new hypervisor image, or extend and capture an existing hypervisor image, you can specify which log files or folders to collect for troubleshooting.
The log files or folders that you specify to be collected for troubleshooting can be collected using one or more of the following ways:
- Collected by the MustGather script package
- Backed up using the log backup plug-in
- Accessed using the log viewer
To specify which log files or folder should be collected for troubleshooting, perform the following steps:
- Create a directory named /etc/logging, if it does not already exist.
- Create a JSON file named <component>.json where component is a unique name within /etc/logging to help identify which log files and folders to collect.
- Edit the <component>.json file to contain the list of files, folders, or folders with patterns that should be monitored for log files for a specific role, for example, operating system, middleware, or other. The following example provides a listing for a role named "OS" where a specific log file named abc.log, everything recursively in the /opt/logs folder, and only files with the .log file extension are monitored.
{ "role":"OS", "types": [ { "logtype": "SingleLine", "type": "file", "name": "/log/abc.log" }, { "logtype": "File", "type": "dir", "name": "/opt/logs" },{ "logtype": "File", "type": "dir", "name": "/opt/specificlogs", "pattern":"*.log" } ] }See the following table for a description of the values that you can use for logtype:
Logtype name Logtype details File Single file created for log entry BinaryFile Single binary file created for log entry SingleLine Each line is a new log entry MultiLineTimeStamp Single/Multi line entry where square bracket encased date/time [10/8/10 16:42:54:109 EDT] notes the start of a new entry. "start": "\\[\\d{1,2}/\\d{1,2}/\\d{2}.*\\d{1,2}:\\d{2}:\\d{2}:\\d{3}.*\\w{1,3}\\].*"
MultiLineIP Single/Multi line entry where ip address notes the start of a new entry "start": "\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}.*"
- To ensure that the log collection is working as expected, run the MustGather script package on a deployed virtual system using the hypervisor image. The output .zip file should contain the logs and folders that you specified if they exist on the deployed system.
4.3. Diagnosing Advanced Middleware Configuration problems with message logs
The topics contained in this section provide information that is intended to help you diagnose and solve problems that you might encounter with Advanced Middleware Configuration using message logs and trace information.
4.4. Monitor LED status
Use the system console to monitor the LED status for hardware in the system.
You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps.
Procedure
- Access the console pane. Click System Console > System > Troubleshooting.
- Expand LED status. A list of processors is provided in a table, and each processor is associated with a compute node. The table also includes the status of the LEDs.
- Filter the data. Select the status of the LEDs to filter the processors that are displayed in the table. In the Status menu, select All, Off or On. By selecting All, you can generate a table that includes all processors that have LED status on or off. By selecting Off, you can generate a table that includes all processors for which LED status is Off. To list processors with the LED status On, select On from the menu.
- View the compute node source. Click the <compute_node> link in the Source column of the table.
You filtered the LED status for processors and viewed the compute nodes associated with those processors.
4.5. Monitor system events
You can set filters, such as event type, severity, and category, to refine the list of system events that are displayed. Narrowing the list of events is helpful when you want to troubleshoot issues for a certain device or component in the system.
You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps. You can also access events from the tree view of the infrastructure map. The Infrastructure Map pane default is the graphics view. To ensure that you are on the tree view, select Show Tree View.
Procedure
- Click System > Events.
- Filter the events.
- To filter by event type, select one or more of the following values in the Type field. If you do not specify a filter value, all system events are displayed.
- Select All
- Chassis Management Module
- Chassis Network Switch
- Chassis Node
- Chassis Power Module
- Chassis San Switch
- Chassis Switch
- Compute Node
- LDAP
- Management Node
- PureApplication Service
- PureApplication Systems Manager
- Storage Subsystem
- Systemlogs
- Virtual Machine
- Virtual Management Instance
- Virtual Systems Manager
- Workload Monitoring
The values are not static. The values are returned from the server side and not restricted to the listed values. Also, in some cases, you might see values that begin with Unknown. These values indicate that the server has either located events from a component that is not yet known or is still displaying events from a component that has been recently replaced but not yet updated on the server.
When you are viewing events, note the following limitations:
- New events that are introduced in the current release are not aggregated with existing events from releases earlier than PureApplication System Version 1.0.0.4.
- The event text is displayed in English only.
- Select from the following values in the Severity menu. You can select more than one value. After you select the type and severity, the list is generated automatically.
- Select All
- All event messages generated by the system.
- Fatal
- Event messages that will likely result in an immediate failure of the system.
- Critical
- Critical events require assistance from IBM support. However, make sure to understand the criteria for deciding how urgently to contact IBM support. With some critical events, the machine can continue to operate successfully due to the redundancy of the hardware, the networking and overall architecture. If a critical event occurs repeatedly, which is indicated by the count field, contact IBM support and gather the appropriate logs.
- Major
- Event messages that can or have affected the system, and should be addressed with high priority.
- Minor
- Event messages that can or have affected the system, and can be addressed with lower priority.
- Warning
- Warning events provide useful information within the context of the system, and are often used by IBM support to diagnose problems that are identified by failures, as well as critical and call support events. These events can reflect situations that might eventually lead to a critical event if the automated recovery steps fail. The system should recover, but warning events provide a breadcrumb trail for IBM support if required.
This type of event represents state changes within the system, but is not necessarily a cause for concern or action. For example, the Image Construction and Composition Tool sends an event when the tool is stopped, which can impact to other users of the system. The system administrator can track this event to identify why the workload was stopped. You can use the user interface as described in step 2 to filter these events and identify more serious events that might be occurring within the system.
- Informational
- Informational events provide useful information within the context of the system, and are often used by IBM support to diagnose problems that are identified by failures, as well as critical and call support events.
This type of event represents state changes within the system, but is not necessarily a cause for concern or action. For example, the Image Construction and Composition Tool sends an event when the tool is stopped, which can impact to other users of the system. The system administrator can track this event to identify why the workload was stopped. You can use the user interface as described in step 2 to filter these events and identify more serious events that might be occurring within the system.
- Audit
- Events that are the result of system auditing.
- Debug
- Messages that are displayed when other errors are being debugged.
- Unknown
- Event messages from a device that does not have a severity associated with it. These event messages are typically the lowest type of severity.
- Select the following category types from the Category menu.
- All
- Alert
- Resolution
- Call support: Events of this category can have an associated critical severity, which indicate that something has failed and requires urgent attention. Call support events that are not critical represent a piece of hardware that has detected a failure but will continue working for a period of time. Even though attention to these events is less urgent, they can result in a critical failure and the hardware might become inoperable if ignored. As a best practice, still contact IBM support for any Call support event.
- Customer serviceable: Events of this category can include required physical checks, for example CMM event occurs when compute node temperatures differ. If the physical checks do not uncover an environmental problem that you can resolve, contact IBM support for further assistance with collecting the correct hardware logs and resolving the problem.
- Select the following options from the Time interval menu.
- All
- Shows all events
- Last Hour
- Shows the events that occurred in the last hour
- Last 24 Hours
- Shows the events that occurred in the last 24 hours
- View the event count. The count displays how many times the event occurred. Click the number to show the detailed timestamp. Click the arrow in the Count column header to show the most frequent event at the top of the list.
Contact IBM support for assistance with recurring events even if you do not observe failures with workloads or other functions.
- Add a comment on an event. You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps. With this permission, you can only comment on events related to hardware.
- Click the Comments icon in the Actions column.
- Type a comment in the field.
- Click Post a comment.
- Click OK.
- Delete an event from the list. Click the Delete icon in the Actions column. To delete more than one event at the same time, select the check boxes beside the events, and click the red X icon above the Actions column. To delete all events or all events in the current filter, select the check box at the top of the event list next to the Event Text column title. Then click the red X icon.
- View the event details. Click the Show details icon in the Actions column. If you open a PMR related to an event, ensure that you include the information from the event details page.
4.6. Monitor system problems
Use the system console Problems pane to monitor and filter problems that occur in IBM PureApplication System W1500.
You must be assigned the Hardware administration role with permission to View all hardware resources (Read-only) to perform these steps. This pane lists all of the problems that exist in the system. You can filter the problems by status or failed machine. Included in the table are problem text to describe the problem, the machine that failed, problem details, status such as open and closed, and the date that the problem arose. Any event that is marked as Call support is added to the table. These events represent issues that require assistance from IBM support.
Procedure
- Click System > Problems.
- Filter the system problems by status. Click the Status menu and select from one of the following: All, Open, or Close.
- Filter the status by failed machine. Enter the name of the failed machine in the Failed machine field.
- Review the details for each problem by clicking the Show details icon in the Action column.
- Export a list of system problems to a file that you can download to your local system. Click the Export icon and Save.
- Delete a problem by clicking the Delete icon in the Action column.
4.7. Manage trace settings
As you troubleshoot issues with the system, add or modify trace settings, the details of which you can download in log files, to effectively diagnose and solve problems.
You must be assigned the Hardware administration role with permission to Manage hardware resources (Full permission) to perform these steps.
You can view and download two log files: error.log and trace.log. Each log file has a maximum size that it cannot exceed and a file limit defining the number of versions of that file that can be maintained on the system. Each error.log file can be a maximum of 2 MB in size and up to five versions of the file are maintained for a total of 10 MB of available data. Each trace.log file can be a maximum size of 100 MB in size and up to 20 versions of the file are maintained (1 active file and 19 archived files). If a version of either log file is older than 30 days, it is automatically deleted.
A set of default classes is defined as the trace string to be included in the logs. The level of trace for these classes can be modified and new classes can be added. The trace levels provided are based on Java. Logging convention and WebSphere Application Server levels.
Procedure
- Click System > Troubleshooting.
- Expand the Trace Setting section.
- Modify the existing trace levels. Click the current trace level setting and select the new trace level from the available options, which are listed as follows:
- OFF
- No events are logged
- SEVERE
- The task cannot continue, but the component can still function
- WARNING
- Potential error or impending error
- INFO
- General information outlining overall task progress
- FINE
- Trace information - General trace
- FINER
- Trace information - Detailed trace + method of entry /exit / return values
- FINEST
- Trace information - A more detailed trace - Includes all the detail that is needed to debug problems
- ALL
- All events are logged. If you create custom levels, ALL includes your custom levels and can provide a more detailed trace than FINEST.
- Add new trace settings.
- Scroll to the bottom of the list, and click Add trace setting.
- Enter the name of the trace setting in the field.
- Select the trace level setting. By default, the trace level is set to INFO.
- Click OK to save your changes.
- Delete trace settings by clicking the X icon.
4.7.1. Configure tracing for the workload agent
You can enable tracing for the workload agent process to assist with debugging and troubleshooting virtual applications. By default, only information level messages are recorded. When you need more detailed trace logs, you can enable all log levels. There are two ways to enable tracing for the workload agent.
- Configure the workload agent plug-in to enable tracing of the agent process for all new virtual application deployments.
- Enable tracing for the agent in an existing virtual application instance
Configure tracing for all new deployments
Use these steps to enable tracing for the workload agent in all new virtual application deployments.
Note: When detailed tracing is enabled for the workload agent, the performance of virtual applications might be impacted. When you no longer require tracing to troubleshoot, reset the trace string as described at the end of this procedure.
- Click the Workload Console tab at the top of the Welcome page to open the workload console.
- Click Cloud > System Plug-ins.
- Select a Foundation Pattern Type from the list, and then select the agent plug-in.
- Click Configure on the toolbar.
- Specify the appropriate settings
- Trace string applied to the agent
- Type the trace string.
- The default trace string is *=info=enabled. This setting records only info level messages, which are saved in the trace log.
- Use the following string to enable all traces. Enter the trace string as one continuous string without spaces or line breaks.
*=info=enabled:com.ibm.maestro.common.*=all=enabled:com.ibm.maestro.agent.*=all=enabled:com.ibm.maestro.profile. *=all=enabled:com.ibm.maestro.security.*=all=enabled:com.ibm.purescale.profile.*=all=enabled:com.ibm.purescale.security. *=all=enabled:osgi=all=enabled:com.ibm.ws.bbson.ConfigOracle=all=enabled:com.ibm.ws.bbson.api.connectivity.local. *=all=enabled:com.ibm.ws.bbson.api.traffic.local.*=all=enabled:com.ibm.ws.bbson.api.connectivity.remote. *=all=enabled:com.ibm.ws.bbson.api.traffic.remote.*=all=enabled:com.ibm.ws.bbson.BBFactoryImpl =all=enabled:com.ibm.maestro.elbservice.management.*=all=enabled:com.ibm.maestro.sharedservice.genericREST.resources. *=all=enabled- Trace File Name
- The name for the trace file. The default file name is trace.log.
- Trace Max File Size
- The maximum trace log file size in MB. The default size is 40 MB. When a log file reaches the specified maximum size, a new log file is created. The file name for the new trace log includes a numeric suffix, and the suffix increments for each new file. For example, trace.log.1, trace.log.2, trace.log.3.
- Trace Max File Number
- The maximum number of trace log files that can be created. The default is 20 files. Each trace log file has a numeric suffix. For example, trace.log.0, trace.log.1, trace.log.2, trace.log.3. If the maximum number of files is reached, a new sequentially numbered trace log file is created and the oldest trace log file is deleted. For example, if the maximum value is 20, only 20 trace log files are saved at a time. The file trace.log.20 is created when trace.log.19 reaches the maximum file size and trace.log.0 is deleted.
- Append to trace file
- If this option is selected, data is appended to the most recent existing trace log file each time a new set of information is recorded to the trace log until the maximum file size is reached. If this option is not selected, a new trace log file is created each time a new set of information is required in the trace log, even if the maximum file size of the most recent log file is not reached.
- Click Update. Tracing is enabled for all the workload agents in new deployments.
- After you deploy a virtual application, you can view the trace logs for the agent. Perform the following steps.
- Click Instances > Virtual Applications.
- Select a virtual application instance.
- Click Manage on the toolbar.
- Click the Logging tab. Logs are organized by virtual machine name, and each name contains the name of the roles that are associated with the virtual machine. A role within a virtual application refers to the software that is installed and configured.
- Expand the IWD Agent role and then expand the entry with the /logs subdirectory. For example, if the virtual machine name is Web_Application-was.11332975226124, expand the entry Web_Application-was.11332975226124/logs.
The log file with the file name that you configured is listed under the /logs directory. Select the trace log to view the contents in the detail pane.
- When you no longer need the trace logs for troubleshooting or debugging, reset the trace string. Repeat the preceding steps and set the trace string as follows:
*=info=enabled
Configure tracing for a specific virtual application instance
If you need detailed trace logs for the workload agent in an existing virtual application instance and tracing of all log levels was not configured at deployment time, you can use these steps to configure detailed tracing for the workload agent in the virtual application instance.
Note: When detailed tracing is enabled for the workload agent, the performance of virtual applications might be impacted. When you no longer require tracing to troubleshoot, reset the trace string as described at the end of this procedure.
- Click the Workload Console tab at the top of the Welcome page to open the workload console.
- Click Instances > Virtual Applications.
- Select a virtual application instance.
- Click the Manage icon on the toolbar.
- Click the Operation tab.
- Select the Agent role from the list operations. The detail pane displays configuration options.
- Expand Update trace settings and specify the appropriate settings.
- Trace string applied to the agent
- Type the trace string.
- The default trace string is *=info=enabled. This setting records only info level messages, which are saved in the trace log.
- Use the following string to enable all traces. Enter the trace string as one continuous string without spaces or line breaks.
*=info=enabled:com.ibm.maestro.common.*=all=enabled:com.ibm.maestro.agent.*=all=enabled:com.ibm.maestro.profile. *=all=enabled:com.ibm.maestro.security.*=all=enabled:com.ibm.purescale.profile.*=all=enabled:com.ibm.purescale.security. *=all=enabled:osgi=all=enabled:com.ibm.ws.bbson.ConfigOracle=all=enabled:com.ibm.ws.bbson.api.connectivity.local. *=all=enabled:com.ibm.ws.bbson.api.traffic.local.*=all=enabled:com.ibm.ws.bbson.api.connectivity.remote. *=all=enabled:com.ibm.ws.bbson.api.traffic.remote.*=all=enabled:com.ibm.ws.bbson.BBFactoryImpl =all=enabled:com.ibm.maestro.elbservice.management.*=all=enabled:com.ibm.maestro.sharedservice.genericREST.resources. *=all=enabled- Trace File Name
- The name for the trace file. The default file name is trace.log.
- Trace Max File Size
- The maximum trace log file size in MB. The default size is 40 MB. When a log file reaches the specified maximum size, a new log file is created. The file name for the new trace log includes a numeric suffix, and the suffix increments for each new file. For example, trace.log.1, trace.log.2, trace.log.3.
- Trace Max File Number
- The maximum number of trace log files that can be created. The default is 20 files. Each trace log file has a numeric suffix. For example, trace.log.0, trace.log.1, trace.log.2, trace.log.3. If the maximum number of files is reached, a new sequentially numbered trace log file is created and the oldest trace log file is deleted. For example, if the maximum value is 20, only 20 trace log files are saved at a time. The file trace.log.20 is created when trace.log.19 reaches the maximum file size and trace.log.0 is deleted.
- Append to trace file
- If this option is selected, data is appended to the most recent existing trace log file each time a new set of information is recorded to the trace log until the maximum file size is reached. If this option is not selected, a new trace log file is created each time a new set of information is required in the trace log, even if the maximum file size of the most recent log file is not reached.
- Click Submit. Tracing is configured for all the workload agents in the deployment.
- Use the following steps to view the trace logs for the agent.
- Click the Logging tab. Logs are organized by virtual machine name, and each name contains the name of the roles that are associated with the virtual machine. A role within a virtual application refers to the software that is installed and configured.
- Expand the IWD Agent role and then expand the entry with the /logs subdirectory. For example, if the virtual machine name is Web_Application-was.11332975226124, expand the entry Web_Application-was.11332975226124/logs.
The log file with the file name that you configured is listed under the /logs directory. Select the trace log to view the contents in the detail pane.
- When you no longer need the trace logs for troubleshooting or debugging, reset the trace string. Repeat the preceding steps and set the trace string as follows:
*=info=enabled
Troubleshoot by symptom
Troubleshoot the system by symptom when there is limited event or error code information or when you observe problems that are not reflected in the system logs.
5.1. CMM event occurs when compute node temperatures differ
A warning message is displayed when the values between the compute nodes that report the highest and lowest temperatures differ by 5 degrees Celsius.
Symptoms
When there is a difference of 5 degrees Celsius between the compute node that reports the highest temperature and the compute node that reports the lowest temperature for a particular chassis, the following warning message is displayed in the events log:
40050000: Hot air exiting from the rear of the chassis might be recirculated in the inlet air at the front of the chassisCommon causes for this event are due to either cold or hot air escaping from beneath the system and reaching the lowest ambient temperature sensors. In some cases, the current system might be properly sealed, but the adjacent system can have openings that allow hot air from the exhaust aisle to recirculate to the front of the system.
Resolving the problem
- Check the physical locations of the hottest and coldest ambient temperature readings.
- Check the chassis temperature. In the system console, click Hardware > IBM Flex Management Chassis > chassis_name.
- Confirm the sill-plate/rack-skirt/tilt-plate/baffle is installed at the front edge of the bottom of the system.
- Look for any openings that might permit the system.s hot exhaust air to recirculate to the front of the system. This can range from possible cable raceway end caps being removed to missing filler or blank panels between the servers and the switches.
- Check the event log for other messages regarding the temperature of the chassis or other events. In the system console, click System > Events.
5.2. Troubleshoot LDAP connection issues
Use this information to troubleshoot possible Lightweight Directory Access Protocol (LDAP) connection issues.
During the LDAP authentication process, general authentication or internal authentication failures might occur, that can prevent a successful login. General authentication failures are due to incorrect user name and password entries. Internal authentication failures can be caused by any of the following errors:
- connection error
- connection timeout
- filter syntax error
- search attribute error
- communication error
- resource shortage error
In any case, the following message is displayed:
Your user name or password is not valid.When a user reports that they cannot log in to the system, even though they are typing the correct password, the administrator can review system events to find out what is causing the authentication issue.Note: Numerous types of events might be recorded in the system log. To only view LDAP events, sort them by changing the Type field to LDAP. If LDAP does not appear in the list, that means that no LDAP connection issues were detected. The following is an example of what might be displayed in the log:
CWZIP4665W: The connection to LDAP has failed. The following error occurred: CommunicationException: 172.16.248.10:389This message communicates that an error occurred during a connection attempt to the LDAP server, which might be due to an incorrect PureApplication System login. It can also be caused by a mis-configured parameter under the LDAP settings tab. To troubleshoot LDAP connection failures, complete the following procedure:
Procedure
- Review the event details for the exception that was caught. To access the system event logs, click System > Events. When you receive an internal error during LDAP authentication, an event entry is created of the type .LDAP.. The following example shows a possible exception:
CommunicationException: Connection Timed Out- Ensure that the LDAP settings are correct. To verify, click System > Security.
Note: If a login failure is reported, and the event log does not contain an entry specifying that the connection to the LDAP server has failed, then the log in failure is more likely to be a general authentication issue.
A resolution event entry is created on the Events page, that informs you that the connection to the LDAP server has been restored, following an internal error with the LDAP server. The event is generated only once a successful request to the LDAP server has been made. You might see a message similar to the following example:
CWZIP4666I: The connection to LDAP has been restored.
5.3. Hypervisor errors and scenarios that can cause them
You can receive error messages for hypervisors defined to the system under certain circumstances.
If you change a password on a started hypervisor, the system gives you error messages. The following error messages are shown for each of the following scenarios:
- Login is not valid
- This message is shown, in the user interface, on the Hypervisors window if you have performed the following steps:
- Defined a hypervisor to the system and started it.
- Changed the password on the hypervisor, but did not change it in the system.
- Rediscovered the hypervisor in the system.
Be sure that you change the password for the hypervisor on PureApplication System, and not just on the hypervisor itself, to prevent this issue.
- Login is not valid
- This message is shown, in the user interface, on the Hypervisors window and also on the deployment wizard if you have performed the following steps:
- Defined a hypervisor to the system and started it.
- Changed the password on the hypervisor, but did not change it in the system.
- Deployed work to that hypervisor, usually by creating a cloud group with just that hypervisor in it.
Be sure that you change the password for the hypervisor on the system, and not just on the hypervisor itself, to prevent this issue.
- Unable to connect to endpoint. URL: <address_of_hypervisor_that_cannot_be_reached>
- This message is shown, in the user interface, if you have performed the following steps:
- Defined a hypervisor to the system and started it.
- Disconnected the hypervisor from the network.
- Rediscovered the hypervisor in the system.
- No physical machines are found for the hypervisor
- When you receive this message, you might need to reconnect to the VMware Virtual Center. Use the following general procedure:
- If the hypervisors have active deployments, quiesce the hypervisors.
- Put the hypervisors in maintenance mode.
- Rediscover the VMware Virtual Center.
- Re-configure any network or storage options, if they changed because of the rediscover process.
- Start the hypervisors.
If the problem is not resolved, you might need to reboot the hypervisors and then reboot and rediscover the VMware Virtual Center.
5.4. Self-describing capability does not work for an external PureApplication System Agent
Problem: Self-describing capability does not work for an external PureApplication System Agent. When the PureApplication System Agent is installed or upgraded, its application support is not automatically installed or upgraded on the remote monitoring server and hub monitoring server.
Solution: Install the IBM Tivoli Monitoring agent framework V6.2.3 or later on the system where the external PureApplication System Agent is installed.
5.5. Troubleshoot the System Monitoring service
This section contains problems that you might encounter while using the System Monitoring service and solutions that you can use to fix these problems. Use the following information to help you troubleshoot problems related to the System Monitoring service:
5.5.1. KFWITM088W Additional Topology information not available
When opening the PureApplication System Monitoring Portal, the error message "KFWITM088W Additional Topology information not available at this time. The network configuration might be changing at this time. Please try again later." is displayed.
To resolve this problem, restart the Tivoli Enterprise Portal Server.
5.5.2. No monitoring data for the PureApplication System
No monitoring data for the PureApplication System is presented in the workspaces of Tivoli Enterprise Portal.
No monitoring data for the PureApplication System is presented in the workspaces of Tivoli Enterprise Portal. This problem occurs if, for some reason, the PureApplication System is down for over five days and then restarted.
To fix this problem, restart the external PureApplication System Agent ...s:
- cd ITM_install_dir/bin
- ./itmcmd agent -o agentinstance stop q8
- ./itmcmd agent -o agentinstance start q8
where ITM_install_dir is the installation directory of IBM Tivoli Monitoring, and agentinstance is the name of the external PureApplication System Agent instance.
5.5.3. TEPS connectivity error
TEPS connectivity error is displayed in the Summarization and Pruning agent workspace.
The value of the TEPS Connectivity attribute in the Connectivity table of the Summarization and Pruning Agent workspace is No. It will cause historical data to be unavailable in the workspaces.
To resolve this problem, restart the Tivoli Enterprise Portal Server.
5.5.4. System Monitoring service remains in launching state for over an hour
The System Monitoring service might remain in launch state after deployment.
Administrators might notice that deployments that are monitored by System Monitoring service have unknown status. During this time, up or down scaling might stop if these deployments use a scaling policy.
As a workaround, if the System Monitoring service has been in launching state for over an hour, delete and redeploy the service. Deployments that have been monitored by the System Monitoring service will reconnect to the newly deployed service, resulting in known health status. No user intervention or down time for the monitored deployments is expected.
5.5.5. Error occurs when middleware monitoring shared service is deployed without a running System Monitoring shared service
Monitor services for middleware depend on the base System Monitoring service to function.
Administrators can extend the System Monitoring service to collect performance and availability data about middleware applications in a cloud group. Monitoring services for middleware are separate shared services deployed in addition to the base System Monitoring service. Examples of these service include monitoring services for IBM WebSphere Application Server and IBM HTTP Servers.
Monitor services for middleware depend on the base System Monitoring service to function. If deployments that are monitored by these services for middleware fail, confirm that the base System Monitoring service is running. If not, deploy the base System Monitoring service. Alternatively, delete the middleware monitoring service. Then try to launch your deployment again.
5.6. Manually refreshing WebSphere Application Server configuration with the proxy shared service
If an IBM WebSphere Application Server instance fails to register with the elastic load balancer a deployment or update of a virtual application instance, you can manually refresh the configuration. If the Secure Sockets Layer (SSL) certificate for the WebSphere Application Server instance is updated, the elastic load balancer also updates the certificate in the storehouse.
Procedure
- Click the System Console tab at the top of the Welcome page to open the system console.
- Click Instances > Virtual Applications.
- Select the virtual application instance. The virtual application instance details display.
- On the details pane, check the current status of WebSphere Application Server in the Middleware perspective section. If WebSphere Application Server is in a failed state, you can restart the instance.
- Click Manage on the menu. The Virtual Machine Monitoring pane displays.
- Click the Operation tab. The Operation pane displays.
- To recover a failed WebSphere Application Server instance, select it in the list. Under Fundamental settings, expand Start or restart the server and click Submit.
- Select the ELBClient instance. The instance details display.
- Expand Refresh Configuration with Proxy Shared Service.
- Click Submit. The elastic load balancer tries to register WebSphere Application Server instances in the virtual application instance.
- If a WebSphere Application Server instance registers successfully to the elastic load balancer, and the SSL certificate file is not updated, a message is displayed to inform you that a certificate update will not be done.
- If a WebSphere Application Server instance registers successfully to the elastic load balancer, and the certificate file is updated, the certificate file is updated in the storehouse.
- If WebSphere Application Server fails register to the elastic load balancer, the WebSphere Application Server registration configuration is refreshed. Before refreshing, the WebSphere Application Server role was in a FAILED state. After successfully refreshing, the role is in RUNNING state.
WebSphere Application Server instances in the virtual application instance are registered and any updated certificates are stored in the storehouse.
5.7. Environment profile not available for virtual application pattern deployment
When deploying a virtual application pattern, the required environment profile is not offered for selection.
When deploying a virtual application pattern, the environment profile to use might not be displayed for selection. This might occur if the environment profile is configured to use Pattern deployer to obtain IP addresses, instead of IP Groups. Configuring the environment profile to use Pattern deployer is not supported for virtual application patterns.
This problem might also occur if you attempt an automated deployment, specifying an environment profile and IP group. In this case you might encounter the following error message:
IOError: CMPUT0087E: IP source of Environment profile with ID 7 is not provided by IP group.To resolve this problem, modify the configuration of the environment profile to use IP Groups instead of Pattern deployer, and try the deployment again.
For information on configuring the environment profile to specify where IP addresses are to be obtained, see Modifying environment profiles.
5.8. Virtual systems deploy successfully with failed scripts
When you deploy a virtual system, failures with the associated deployment scripts can occur.
Symptoms
The deployment status of a virtual system might be displayed as completing successfully even though one or more non-critical scripts associated with the deployment fail to complete successfully.
Resolving the problem
Be sure to examine the History section of the Virtual System Instance page to ensure that associated scripts completed successfully during deployment. To view the historical details, click Workload Console > Instances > Virtual System Instances > instance_name.
Some script packages that are defined to run at deployment time are critical to the success of a deployment, and can cause the deployment to fail if the script does not complete successfully. There are a few scripts, however, that will not result in a failed deployment if they fail during deployment:
- Must Gather Logs
- Default Add User
- Default Add Nic
- Default AIX add NIC
- WebSphere Application Server Hypervisor scripts
Each script execution has an associated script status. After the script has completed execution, the script status will have one of the following values:
- RM01012 - COMPLETE
- Completed successfully with a return code of 0.
- RM01013 - FAILED
- Completed with a return code other than 0 or 2, or did not get any result back from the virtual machine,
- RM01014 - FAILED
- Script could not be transferred to the virtual machine.
- RM01015 - FAILED
- Script did not complete before the time out expired.
- RM01059 - WARNING
- Completed with a return code of 2.
5.9. Situation events not forwarded to event receivers
If situation events are not forwarded to event receivers, use the information in this topic to solve the problem.
Problem: A situation is fired and an alert icon is displayed on IBM PureApplication System Monitoring Portal, however; the situation event is not forwarded to event receivers.
Solution: This problem occurs if the situation is not configured to forward events to an event receiver, or if the situation is a customized predefined situation, or if the situation is newly created. To fix this problem, you must ensure that forwarding events to an event receiver is enabled for this situation, and then manually associate the situation with the event receivers. You can do it through IBM PureApplication System Monitoring Portal. You can also do it by exporting the situation, changing its definition, and then importing the situation back to the server.To fix this problem through the IBM PureApplication System Monitoring Portal, do the following steps:
- In the IBM PureApplication System Monitoring Portal, click Situation Editor in the main toolbar.
- Select the situation in problem.
- Click the EIF tab.
- Ensure that the Forward Events to an EIF Receiver option is selected.
- Select the event receivers to which you want the situation events to be forwarded from the Available EIF Receivers and then click the left arrow to move them to the Assigned EIF Receivers list.
- Click OK to save the situation definition with your changes and close the situation editor.
You can also fix this problem by exporting the situation, changing its definition and then importing it back to the server. Here we assume that the situation in problem is named test_tec1 and its definition is as below:
Name : test_tec1 Full Name : Description : Type : Windows OS Formula : *IF *VALUE NT_Cache.Copy_Read_Hits_% *EQ 1 Sampling Interval : 0/0:15:0 Run At Start Up : Yes Distribution : Text : Action Location : Agent Action Selection : System Command System Command : *NONE True For Multiple Items: Action on First Item only TEC Severity : Critical TEC Forwarding : Y TEC Destination : 0Do the following steps to fix this problem:
- Export the situation to an XML file:
C:\ODI>tacmd viewsit -s test_tec1 -e c:\test_tec1.xml- Edit the XML file and change the following lines accordingly, to specify a null destination server:
<SITINFO> <![CDATA[SEV=Critical;TFWD=Y;~;"]] > </SITINFO>- Delete the original situation and import the situation back to the server:
tacmd createsit -i c:\test_tec1.xml
Troubleshoot issues with Image Construction and Composition Tool
Use the information and content included here to understand how issues occur during Image Construction and Composition Tool processes.
6.1. Log files for troubleshooting
Check the Image Construction and Composition Tool log files to determine where problems occurred.
To access the Image Construction and Composition Tool server log files, from the Administer menu, click Download logs. Save the compressed file and examine the log files. The compressed file contains the following server log files:
- The trace.log file is the most recent trace file. The default size of the trace.log file is 100 MB, with up to 9 backups.
- The error.log file is the most recent error file. The default size of the error.log size is 2 MB, with up to 4 backups.
To access the log files from the logs directory, go to /drouter/ramdisk2/mnt/raid-volume/raid0/logs and review the contents of the trace and error directories.
If problems occur during Image Construction and Composition Tool installation, check the following Installation Manager log files to troubleshoot installation problems:
/var/ibm/InstallationManager/logs/<date_time>.xml /var/ibm/InstallationManager/logs/ant/<date_time>.logwhere <date_time> is the date and time in a format similar to 20111024_1201.To review specific Image Construction and Composition Tool messages, search in the log files for the lines that start with the message code CYO.
6.2. Log files for virtual image synchronization
If you are synchronizing a virtual image and the deployed VM for the virtual image is created, but the synchronization fails, you can download log files for the synchronization. If the synchronization fails before the deployed VM is created, no log files are available. If the synchronization failed after the deployed VM was created, but before the bundle installation, the following log files are available:
- error.log
- trace.log
If the synchronization fails during or after bundle installation, the following log files are available:
The err.log and out.log files are the log files for the bundle installation during the synchronization process. They are located in the directory:/drouter/ramdisk2/mnt/raid-volume/raid0/local/images/asset_id/logs where asset_id is the asset identifier for the image. To download the log files, complete the following steps:
- error.log
- The most recent error file
- trace.log
- The most recent trace file
- err.log
- The error output of the synchronization script run on the VM, including output of any bundle installation script run during the sync process
- out.log
- The standard output of the synchronization script run on the VM, including output of any bundle installation script run during the sync process
Procedure
- From the Images screen and expand the Virtual System section of the image that you are synchronizing.
- Click Download logs. The compressed file containing the log files is downloaded.
6.3. Adjusting the configuration timeout settings
After you change the configuration timeout settings, you must restart the Image Construction and Composition Tool. You can change the following timeout settings:
# Configurable timeouts "ShimTimeoutInMinutes" : 120, # Enable Single sign-on "EnableSSO" : false, # ICCT instance user "InstanceUser": "", # When executing windows commands how long do we wait before we timeout # in minutes. This default value for this timeout is 3 minutes. "WindowsCommandTimeOut": 3, # When running the execution bundle during a synchronization, # how long do we wait before timing out in minutes. Default # values is 6 hours. "BundleExeuctionTimeout": 360, # IWD REST timeout "restReadTimeout": 90000, # IWD Max deploy wait time (minutes) Default value is 0 which # disables the max Deploy time. If set by default we have put in # 300 minutes (5 hours). If this is set it will work in conjunction # with the following configuration parameters: # IWD_SYSTEM_CREATION_ATTEMPTS # IWD_WAIT_FOR_SYSTEM_CREATION_RETRY_DELAY # IWD_WAIT_FOR_DEPLOY_STATUS_DELAY # IWD_WAIT_FOR_STARTUP_COMPLETE_DEALY # IWD_MAX_SYSTEM_ACTIVATION_WAIT_TIME "maxIWDDeployWaitTime": 0, # IWD Max number of creation attempts also the same # number of attempts for deployment status checks. "IWD_SYSTEM_CREATION_ATTEMPTS": 120, # IWD delay between creation retries (seconds) "IWD_WAIT_FOR_SYSTEM_CREATION_RETRY_DELAY": 30, # IWD delay between checks for deployment status in (seconds) "IWD_WAIT_FOR_DEPLOY_STATUS_DELAY": 10, # IWD wait for startup complete (seconds) "IWD_WAIT_FOR_STARTUP_COMPLETE_DEALY": 30, # Amount of time to wait for the system to become active. This is checked # after system creation and deploy status checks. Default value is 300 # minutes. "IWD_MAX_SYSTEM_ACTIVATION_WAIT_TIME": 300, # How many times to check to see if the capture of an image # has completed before we fail the capture. The default # value is 180 times. "IWD_MAX_CAPUTRUE_ATTEMPTS": 180, # When capturing an images in IWD. How long to wait before # we check the status again in seconds. Default value is 60 seconds. "IWD_WAIT_FOR_CAPTURE_STATUS_DELAY": 60, # Maximum number of time to attempt to capture a VM. # Default values is 240 times. "SCE_MAX_CAPTURE_VM_ATTEMPTS" : 240, # Time to wait between connection attempts in seconds. # Default value is 30 seconds. "SCE_WAIT_BETWEEN_CAPTURE_ATTEMPTS" : 30, # Maximum number of time to try to connect to a VM. # Default value is 120 times. "SCE_MAX_CONNECTION_ATTEMPTS" : 120, # Time to wait between connecting to the VM. # Default value is 10 seconds. "SCE_WAIT_BETWEEN_CONNECTION_ATTEMPTS" : 10, # When running commands on the VM. Time to delay between checking the return # code of the command to see if it is done or not. Default value is 60 seconds "SCE_WAIT_COMMAND_RETRY_DEPLAY" : 60, # Maximum number of time to try and delete a VM. # Default value is 30 times. "SCE_MAX_DELETE_VM_ATTEMPTS" : 30, # Time to wait between checking to see if the VM is deleted. # Default value is 60 seconds "SCE_WAIT_DELETE_VM_DELAY" : 60, # Time to wait after the VM's deletion state goes to deprovision. # Default value is 30 seconds. "SCE_WAIT_DEPROVISION_DELAY" : 30,
6.4. Set logging levels for troubleshooting
When troubleshooting is required, trace logs and error logs are generated. The logged information that is generated depends on definitions that follow the Java Logging convention and WebSphere Application Server levels.
The following log level values, listed from the least amount of logging (OFF) to the most amount of logging (ALL) are available:
- OFF
- SEVERE
- WARNING
- INFO
- FINE
- FINER
- FINEST
- ALL
To modify the data that is logged, use one of the following methods.
Use the command-line interfaceIf you use the command-line interface, you can change the logging levels dynamically. The changes take effect immediately and do not require Image Construction and Composition Tool to be stopped and started. However, these changes are lost after stopping the Image Construction and Composition Tool server. For more information, see the following commands:
icct.trace.spec() icct.trace.set('com.ibm.cloud.icn.execution', 'FINER')
Modify Image Construction and Composition Tool file manuallyCAUTION:
Follow the exact syntax and format when editing this file.
You can update the file <install_dir>/config/logging-default.config manually. These changes will take effect the next time the tool is restarted. To update the file, complete the following steps:
- Change to the directory <install_dir>. For example:
cd /opt/IBM/icct- Create a backup of the following file:
./icn.app/config/logging-default.config- Update the following file according to your requirements:
./icn.app/config/logging-default.config- Stop and restart Image Construction and Composition Tool:
./stop.sh ./start.sh
6.5. VMware ESX cloud provider troubleshooting
You can set a number of logging level values to troubleshoot problems with the VMware ESX cloud provider.
Set the following values:
"com.ibm.cloud.icn.cloudprovider.localesx": "FINEST" "com.ibm.cloud.icn.core": "FINEST" "com.ibm.cloud.icn.execution": "FINEST" "com.ibm.cloud.icn.util": "FINEST" "com.ibm.rainmaker.vmsupport": "FINEST" "com.ibm.venture": "FINEST" "com.ibm.vespa": "FINEST"
6.6. Troubleshoot problems during VM activation
If you encounter problems when activating a VM during deployment or synchronization.
Symptoms
Problems occur when activating a VM during deployment or synchronization.
Resolving the problem
- Linux
- Log on to the VMware vSphere Client console to access the VM.
- Review any log files with the extension .out and .err in the directory /opt/ibm/ae/AR for errors.
- In the directory /opt/ibm/ae/AP, check if a file named noap exists. If it does, complete the following actions:
- Delete the file noap.
- Remove all contents of the directory AR.
- Reboot the system.
- Check that the VM now activates successfully.
If the problem still persists, complete the following steps:
- Check the console as it starts for any indication of failure. In the user interface, you might have to press the Esc key.
- Log in and investigate each activation script, activating each class one at a time. The scripts that are run during activation are listed in the file /opt/ibm/ae/AL/master.al.
For example, if ConfigLocale causes problems, run the following command once:
AE.sh -s ConfigLocaleIf a class fails, run the script command. The file master.al lists which script fails. They are usually in the directory /opt/ibm/ae/AS. Then debug the script, until you find out which line in the script is failing. The next step is to determine why the line in the script failed.- Enter the following command:
df -kand ensure that the file activation.iso is mounted, usually in the directory /tmp.
- Windows
- Log on to the VMware vSphere Client console to access the VM.
- Review any log files with the extension .out and .err in the directory c:\windows\setup\ibm\ar for errors.
- In the directory c:\windows\setup\ibm\ap, check if a file named noap exists. If it does, complete the following actions:
- Delete the file noap.
- Remove all contents of the directory ar.
- Reboot the system.
- Check that the VM now activates successfully.
If the problem still persists, complete the following steps:
- Check the console as it starts for any indication of failure. In the user interface, you might have to press the Esc key.
- Log in and investigate each activation script, activating each class one at a time. The scripts that are run during activation are listed in the file c:\windows\setup\ibm\al\master.al.
For example, if ConfigLocale causes problems, run the following command once:
AE.bat -s ConfigLocaleIf a class fails, run the script command. The file master.al lists which script fails. They are usually in the directory c:\windows\setup\ibm\as. Then debug the script, until you find out which line in the script is failing. The next step is to determine why the line in the script failed.- Enter the following command:
df -kand ensure that the file activation.iso is mounted, usually CD-ROM.
6.7. VMware ESX cloud provider creation fails
Failure during a VMware ESX cloud provider creation.
Symptoms
When you attempt to create the cloud provider, it fails with message: CYOCP0016E: Unable to read cloud provider data specification.
Resolving the problem
The following issues might apply:
- There might be a firewall between the Image Construction and Composition Tool and theVMware ESX machine. Use a web browser to verify that the IP address of the VMware ESX machine is able to be reached from the Image Construction and Composition Tool machine.
- One of the virtual machines on the VMware ESX has network adapters that are not configured correctly. Check the trace.log for the following message: java.lang.IllegalArgumentException: Argument "name" cannot be null. Verify that the virtual machines known to this VMware ESX have network adapters that are correctly configured. Sometimes VPN programs can create network adapters with null names. Remove the incorrect adapter definition and retry.
6.8. Work with enablement bundles
Enablement bundles are bundles that hide cloud-specific details. They provide a base on which you can build virtual images using cloud-independent bundles. When you are building a virtual image, Image Construction and Composition Tool automatically selects an enablement bundle for the virtual image that matches your cloud provider and operating system and version.
The Image Construction and Composition Tool automatically selects an enablement bundle for the virtual image. During the virtual image building process, you cannot remove the enablement bundle from the virtual image. If you are creating a virtual image and the base virtual image already contains an enablement bundle, no action is performed.
When you work with virtual images and bundles, you must keep the enablement bundles in Image Construction and Composition Tool workspace. When you select a base virtual image, it is automatically added to the appropriate enablement bundle. After you extend the virtual image, giving it a name, symbolic name, version, and description, you see the following message in the Operating System section of the virtual image:
Activation Framework: <name of enablement bundle selected by the tool>You can then add bundles to the virtual image and synchronize it.Do not delete the Enablement Bundle for Exploiter Name enablement bundle. This bundle is required.
If the enablement bundle is removed or not available, the following information is displayed in the Operating System section of the virtual image:
Activation Framework: <Not found>If you accidentally delete this enablement bundle, you can import it again from the following location: <installation_path>/icn.app/private/expanded/ibm/icn.enablement-<version>/bundles/R1/com.ibm.icon.enablement.ux.iwd_<version>.ras where <installation path> is similar to /opt/IBM/icon. Access this top-level directory and change directory to the appropriate cloud provider to find the enablement bundle for the particular operating system you require.
6.9. Update Activation Engine in a virtual image
If Image Construction and Composition Tool can successfully deploy a virtual image, Activation Engine is automatically updated. However, you might need to update Activation Engine if the image cannot be successfully deployed by the cloud provider.
There is a script provided with the Image Construction and Composition Tool that you can run to update Activation Engine in virtual images.
Note: This topic describes upgrading from Activation Engine version 2.0 and higher. Upgrading from version 1.x is a more complex procedure that requires more than a simple version upgrade. For more information, see the Activation Engine documentation.
You can check the version of Activation Engine in one of the following ways:
- Display the content of file /opt/ibm/ae/version.
Note: Does not apply to IBM i.
- Call Activation Engine with the --version flag by running ./AE.sh --version. Look for a line similar to: [2012-05-12 23:29:15,510] INFO: 2.1.1.14.
- Use RPM to query the installed version of Activation Engine by running rpm -qa | grep activation-engine.
Note: Does not apply to IBM i.
Some Activation Engine versions between versions 2.1 and 2.1-1.23 delete user data because some RPM scripts are incorrectly called during the upgrade. So, for versions between 2.1 and 2.1-1.23, invoke the RPM with the --noscripts flag to preserve user data.
Automated upgradeTo perform an automated upgrade, run the following script:/opt/ibm/ae/upgradae.sh new-rpm-path where new-rpm-path is the new RPM that upgrades an installed RPM.
Note: Does not apply to IBM i.
Manual upgradeTo upgrade Activation Engine manually, determine the level of the RPM installed. Depending on the level, run the following command:
- RPM prior to version 2.1-1.24
- rpm --noscripts --force -Uvh new-rpm-path
- RPM from 2.1-1.24 and higher
- rpm -Uvh new-rpm-path
Note: Does not apply to IBM i.
6.10. Image Construction and Composition Tool does not respond
Determine the reasons that the Image Construction and Composition Tool might not respond.
6.10.1. Image Construction and Composition Tool does not start after installation
Before running the IBM Installation Manager installer to install Image Construction and Composition Tool on a system, ensure that the port number 443 is not in use by another application. If the port number 443 is not available, Image Construction and Composition Tool does not start after you completed the installation.
Symptoms
The Image Construction and Composition Tool fails to start and a ZSO error occurs.
Causes
Port 443 is in use by another application.
Diagnosing the problem
Check the following Installation Manager log files to troubleshoot the problem:
/var/ibm/InstallationManager/logs/<date_time>.xml /var/ibm/InstallationManager/logs/ant/<date_time>.logwhere date_time is the date and time in a format similar to 20111024_1201.The following error message appears in the Installation Manager log files:
CWPZC8028E: An unknown ZSO error occurred. Launching of the ZSO failed with return code -1 CWPZT0601E: Error: Command start failedResolving the problem
Before installing Image Construction and Composition Tool, ensure that port number 443 is not in use by another application on the system.
6.10.2. Image Construction and Composition Tool returns EOFException
While running the Image Construction and Composition Tool on a system, you receive a 500 error code and an EOFException.
Symptoms
The Image Construction and Composition Tool fails to respond to user requests and EOFExceptions are generated.
Causes
The Image Construction and Composition Tool has run out of disk space.
Diagnosing the problem
The following error appears in the user interface: java.lang.RuntimeException: Unable to create /drouter/ramdisk2/mnt/raid-volume/raid0/.zero/userzoneStore/.
Check the log file to troubleshoot the problem. The following error message appears in the log file:
[2012-06-19 12:42:16:980 EDT] 00000021 JavaGCSeriali E zero.core.context.zones.JavaGCSerializer deserialize CWPZC0029E: Unable to serialize key <[B@7ee67ee6> in the ser zone. java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully (ObjectInputStream.java:2298)Resolving the problem
To resolve the problem, complete the following steps:
- Stop the Image Construction and Composition Tool using stop.sh.
- Delete all .ser files located in the /drouter/ramdisk2/mnt/raid-volume/raid0/.zero/userzoneStore directory and subdirectories.
- Start the Image Construction and Composition Tool using start.sh.
- Free up /drouter disk space by deleting any images that you do not need.
6.11. Install executable launcher error during installation
If you try to install the Image Construction and Composition Tool on a system where it already exists, the installation is not successful.
Symptoms
When you run ./install, the Image Construction and Composition Tool installation fails with the following error message:
The Install executable launcher was unable to locate its companion shared library.Causes
The error indicates that a version of the Image Construction and Composition Tool already exists on the system.
Resolving the problem
Upgrade the Image Construction and Composition Tool instead of installing it.
Note: If you launched the installation from a mounted Windows driver, copy and run the installation locally.
6.12. Exec error 1 during installation
You receive an executable error when you try to install the Image Construction and Composition Tool on a system.
Symptoms
You receive the following error during installation:
'exec error 1'Causes
The error indicates that port 443 is already in use and the Image Construction and Composition Tool cannot access it.
Resolving the problem
Ensure that port 443 is not in use by other applications.
6.13. Package installation error during silent upgrade
In some cases, when upgrading IBM Image Construction and Composition Tool silently, errors might occur.
Symptoms
During the silent upgrade process, the following error might be displayed:
ERROR: There is already a package installed at "/opt/IBM/icon". The installation directory must not be the same as a previously used installation directory.Causes
This error occurs because of some mismatched product names, depending on the version that you are upgrading from and the installation methods that you used to install Image Construction and Composition Tool previously.
Resolving the problem
To resolve the problem, modify the following two lines in the file <extract_dir>/icon_silent_install_response_file.xml, by changing "Tool" to "Tools" in both cases:
<profile id='Image Construction and Composition Tool' installLocation='/opt/IBM/icon> <offering id='com.ibm.cloud.icon.troy' profile='Image Construction and Composition Tool' features='icon_core' installFixes='none'/>
6.14. Incorrect synchronization parameters after Image Construction and Composition Tool upgrade
Images imported in older versions of Image Construction and Composition Tool do not have correct synchronization parameters.
Symptoms
If you imported images in a version of Image Construction and Composition Tool earlier than V1.2.0.2, they do not have the correct synchronization parameters after the upgrade.
Resolving the problem
When upgrading to Image Construction and Composition Tool V1.2.0.2, a deployment filtering function is enabled to display only the environment profiles and cloud groups that are compatible with the image to be synchronized. This filtering function is enabled only for images imported and extended after V1.2.0.2 is installed. Images that were imported or extended before the V1.2.0.2 installation do not have this function.
6.15. Universal ID and version must be unique
The combination of universal ID and version for every asset must be unique. If you extend an image and you do not enter a unique universal ID and version number combination for the newly extended virtual image, you receive an error message.
Symptoms
When you are extending a virtual image, the following error message is included in the trace.log and error.log files:
Cannot create the image: Error: HTTP response code:409 message: "CYOIG0005E: Universal ID '<ID>_<version>' is not unique."Causes
The universal ID in combination with the version for the new virtual image that you are extending must be unique.
Resolving the problem
Enter a unique universal ID and version combination for the extended asset.
6.16. Cancel - use the current file link does not work
The Cancel - use the current file link, located on the Files to Copy section on the Install or Config tab for a software bundle, might not work correctly.
Symptoms
This problem might occur if you upload a file to the Files to Copy section on the Install or Config tab for a software bundle, save that file, and edit it. After saving and editing the file, you then select a new file to upload and click Cancel - use the current file. Although you tried to revert to the previous file, the new file is still displayed.
Resolving the problem
If the file is not being updated correctly, click Cancel . If you click OK, and see the wrong file, refresh the software bundle to omit all the changes.
6.17. VMware ESX error if you use virtual machines with snapshots
If you use virtual machines with snapshots in VMware ESX, you receive an error in the trace log file. VMware ESX cloud provider does not support virtual images where the virtual machine has snapshots.
Symptoms
You receive an error message in the trace log file that contains output similar to the following:
java.io.IOException: disk conversion of /drouter/ramdisk2/mnt/ raid-volume/raid0/templates/2/base-rhel55-64-000001-delta.vmdk from monolithicFlat to streamOptimized failed!In the preceding example, the -delta.vmdk syntax indicates that there are snapshots in the virtual machine.Causes
The VMware ESX cloud provider does not support images in which the virtual machine has snapshots.
Diagnosing the problem
You receive an error message similar to the following in the trace.log file:
[2011-10-17 17:50:52:509 EDT] 00001198 IconImageExpo I com.ibm.cloud.icn.execution.engine.IconImageExporter compressMonolithicFlatToStreamOptimizedDisk About to convert monolithicFlat router/ramdisk2/mnt/raid-volume /raid0/templates/2/base-rhel55-64-000001-delta.vmdk to streamOptimized disk /drouter/ramdisk2/mnt/raid-volume /raid0/tmp/exportedImages/a0e04b2c-54c6-4b84cebed0e2a37 /base-rhel55-64-000001-delta.vmdk[2011-10-17 17:50:52:691 EDT] 00001198 IconImageExpo E com.ibm.cloud .icn.execution.engine.IconImageExporter compressMonolith icFlatToStreamOptimizedDisk Disk conversion failed for /droudisk2/mnt/raid-volume/raid0/templates/2/base-rhel55 -64-000001-delta.vmdk exited with 1[2011-10-17 17:50:52:693 EDT] 00001198 IconImageExpo E com.ibm.cloud .icn.execution.engine.IconImageExporter compressMonolith icFlatToStreamOptimizedDisk disk conversion of /drouter/ ramdt/raid-volume/raid0/templates/2/base-rhel55-64-000001 -delta.vmdk failed[2011-10-17 17:50:52:693 EDT] 00001198 IconImageExpo E com.ibm.cloud.icn.execution.eng ine.IconImageExporter convertToStreamOptimized Unable to convert file /drouter/ramdisk2/mnt/raid-vaid0/templates /2/base-rhel55-64-000001-delta.vmdk from monolithicFlat to streamOptimized.java.io.IOException: disk conv ersion of /drouter/ramdisk2/mnt/raid-volume/raid0/templat es/2/base-rhel55-64-000001-delta.vmdk from monolithicFlat to streamOptimized failed!at com.ibm.clou d.icn.execution.engine.IconImageExporter.compressMonolith icFlatToStreamOptimizedDisk(IconImageExporter.java:582)> at com.ibm.cloud.icn.execution.engine.IconImageE xporter.convertToStreamOptimized(IconImageExporter.java: 413)at com.ibm.cloud.icn.execution.engine .IconImageExporter.run(IconImageExporter.java:238) at java.lang.Thread.run(Thread.java:736)Resolving the problem
To avoid this problem, do not use virtual machines that have snapshots. Remove the snapshots from the virtual machine before using the virtual machine in IBM Image Construction and Composition Tool.
6.18. Failure during OVA disk conversion
Conversion of disk files contained in the Open Virtualization Appliance (OVA) files fails.
Symptoms
During OVA import or OVA export operations, errors occur when the disk files are converted.
Resolving the problem
Verify that the following conditions are met:
- Verify the available disk space in the directory /drouter. The amount of free space must be at least twice the size of the disk being converted.
- Ensure that the required ld-linux shared library is available. If you are getting errors similar to the following example:
Error messages such as "Bad ELF interpreter"verify that the required software is installed. For example, on Red Hat Enterprise Linux V6.1, you must install the following packages:yum install libz.so.1 yum install ld-linux.so.2
6.19. Import the same OVA image into multiple VMware ESX cloud providers fails
Symptoms
During OVA import, error message CYOES0033E is generated. For example:
CYOES0033E: Unable to import image. Another image exists with the name: CoC_RHEL62_64 and version: 1.0.6. If the image has already been imported using another cloud provider, use the Import from cloud provider option instead of the Import option.Resolving the problem
Verify that the image has not already been imported for this VMware ESX cloud provider. If the image has already been imported for a different VMware ESX cloud provider, you can import the image into additional VMware ESX cloud providers. Instead of using the Import option, use the Import from cloud provider option.
6.20. Import or export of the OVA file fails
Symptoms
During OVA import or OVA export operations, errors occur. The log has the following message:
java.io.IOException: No space left on device.Resolving the problem
Ensure that the following conditions are met:
- Verify the available disk space in the directory /drouter. In some cases, the amount of free disk space must be at least twice the size of the image being converted.
- Ensure that unneeded images are deleted from the Image Construction and Composition Tool to allow for more disk space.
- Ensure that older archived log files are removed to allow for more disk space.
6.21. Import a virtual machine fails
Import fails when you are importing a virtual machine.
Symptoms
During virtual machine import using the ESX cloud provider an error occurs. The log has the following message: CYOES0002E: Unable to establish a connection to #.#.#.#
Resolving the problem
Ensure that the following conditions are met:
- You can ping the IP address from the Image Construction and Composition Tool machine.
- The user ID and password are correct.
- No firewall is limiting traffic over port 22 (SSH) or port 445 (Windows).
- The Image Construction and Composition Tool machine has a correct host name, domain and IP address, for example, /etc/hosts contains a correct entry.
6.22. Error occurs during OVA export
Symptoms
During OVA export operations, errors occur. You might see the following error message in the log files:
CYOCU0005Resolving the problem
Ensure that the user specified in the user interface has write permission for the specified target folder.
6.23. OVA deployment to VMware fails with unsupported element 'Property'
If you are attempting to use VMware vSphere Client to deploy an OVA file, the operation fails.
Symptoms
When deploying an OVA template from VMware vSphere Client by using File > Deploy OVF Template, you get the following error message:
This OVF package uses features that are not supported when deploying directly to an ESX host.The additional text of the message states:The OVF package requires support for OVF Properties" and "Unsupported element 'Property'.Causes
OVA deployment directly to ESX or ESXi is not supported. You must use VMware vCenter Server (VirtualCenter). To verify the server type, use VMware vSphere Client to select the server in the left column. In the right window, above the Summary tab, you check whether the server is VMware ESX/ESXi or VMware vCenter Server.
Resolving the problem
Install VMware vCenter Server (VirtualCenter) and deploy the OVA file to this VMware vCenter.
6.24. The script package maestro is in failed state
When you deploy an image that is created with IBM Image Construction and Composition Tool, you might see that the script package maestro is in a failed state.
Symptoms
After you deploy the image, the log file remote_std_err.log contains the following error message, which indicates that a prerequisite is missing:
Verifying Prerequisites - Java ERROR: Missing IBM SDK for Java at /opt/IBM/ibm-java-i386-60Causes
The required package is missing because IBM Image Construction and Composition Tool does not automatically install the IBM SDK for Java during image extension.
Resolving the problem
If the virtual application features provided by the maestro script package are needed, you can extend the image by using IBM Image Construction and Composition Tool and by adding a bundle for IBM SDK for Java.
6.25. Virtual image capture fails
If the virtual image capture fails, you might need to delete the failed virtual image and then create the virtual image and complete the capture process again.
Symptoms
The virtual image capture fails with the following error message:
CYOCE0002E: The capture cannot be started because the resources are not sufficient to capture the image. Check your private instance quota. The image cannot be saved.Resolving the problem
To enable the capture option again, expand the virtual image capture error and click the option to try to capture again. Alternatively, delete the failed virtual image and create the virtual image again. Then, complete the capture process again.
If the problem is not resolved, check the image quota of the IBM SmartCloud Enterprisecloud provider. Log on to IBM SmartCloud Enterprisevia a web browser and check how many stored images you are allowed versus how many you have. You might need to delete a stored image or increase the quota for the number of stored images.
6.26. Virtual images that contain old versions of the activation engine cannot be deployed
If you are attempting to deploy virtual images by using Image Construction and Composition Tool, the deployment of these images might stop unexpectedly.
Symptoms
When deploying virtual images in Image Construction and Composition Tool, if the activation engine version in these images is earlier than 2.1.1.24, the deployment of these images might stop unexpectedly.
Causes
If the version of the activation engine used in the image is earlier than 2.1.1.24, the file ovf-env.done might not be generated and the process cannot detect that the activation completed.
Resolving the problem
Upgrade the automation engine.
6.27. Files are not replaced correctly
If you add a file to the Files to Copy section on the Install, Config, or Rest tab of a software bundle and later replace that file, the content of the file is not updated, even though the name of the file is updated correctly in the Image Construction and Composition Tool. The content is updated correctly after you save the software bundle.
Symptoms
If you replace the file in the Files to Copy section on the Install, Config, or Resttab of a software bundle, the content of the file is not updated unless you save the software bundle. However, the name of the new file is displayed in the Image Construction and Composition Tool.
Resolving the problem
Save the software bundle before you click the file to download the contents.
6.28. The user interface stops responding while uploading files
Image Construction and Composition Tool stops responding when uploading a file that is larger than 2 GB.
Symptoms
When you create a software bundle and upload a file with a size larger than 2 GB in the Install section. You might be waiting a long time for the file to upload, while the user interface seems to stop responding.
Causes
There is a 2-GB file size limit for uploading in the browser. There is little information in the trace and error log files.
Resolving the problem
To upload large files to a software bundle:
- In the New file dialog box, select the Remote option and specify the URL for the file to upload in the text box. Uploads done on this dialog box are asynchronous, while local uploads are synchronous.
- Verify that the file is uploading by checking the status indicator for the entry in the Files to copy table.
- Continue to refresh the software bundle definition until the upload is completed.
6.29. Limitation when performing concurrent updates
Do not perform concurrent update operations on images, bundles, and other objects in Image Construction and Composition Tool, either using different browsers or using a browser and the command line interface.
Symptoms
The object might not be updated correctly.
Resolving the problem
Only use a single mechanism of updating an object in Image Construction and Composition Tool, either using the web interface in a single browser, or using the command-line interface.
6.30. Cannot remove software bundles from a personality
If a software bundle is added to a personality and then the virtual image is synchronized, the added software bundles cannot be removed from the personality.
Symptoms
The software bundles cannot be removed from the personality.
Resolving the problem
Delete the existing personality and create another personality where you add only the software bundles that you need.
6.31. Bundles are not displayed in the list of compatible images
Symptoms
Bundles are not displayed in the list of compatible images.
Resolving the problem
On the Add software bundles dialog box, clear the check box Show only bundles compatible with the image.
6.32. Limitation when adding bundles to a personality
For virtual images that have a single personality, all software bundles that are added to the image are automatically added to the personality. This behavior means that, in this case, the personality is identified with the image itself.
Symptoms
For virtual images that have multiple personalities, software bundles that are added to the image are not added automatically to any of the personalities in the image. For these images, you must explicitly add the software bundles to the personalities, as needed.
Resolving the problem
If you want to avoid this limitation, complete steps in the following order:
- Extend the virtual image with the single personality.
- Add software bundles, as required. The bundles are added to the personality by default.
- Add another personality to the image.
- Add more bundles to the images. The bundles are not added to any personality by default.
- If required, add the bundles to the appropriate personality.
6.33. Bundles not added to new images
Bundles are not added to new images in Image Construction and Composition Tool.
Symptoms
When you are creating an image from a template, bundles without an installation step defined are not added to the new image.
Resolving the problem
After you create an image from a template, add more bundles to that image
6.34. Limitation when deleting images that are synchronizing
You cannot delete an image while it is synchronizing.
Symptoms
The image is synchronizing and the synchronization is not proceeding successfully.
Resolving the problem
Do not delete an image while it is synchronizing. Wait until the synchronization process has completed and then delete it.
6.35. Time lag when deleting virtual images
If you delete a virtual image, it might take awhile for the virtual image to be completely removed from view.
Symptoms
A virtual image that you deleted is still displayed in a view in the tool. You can view and select the virtual image for some time until the deletion process has completed.
Causes
There is a refresh issue that causes a deleted virtual image to remain in view until the deletion process has completed. The deletion process can take awhile to complete.
Resolving the problem
Wait until the deletion process has completed and refresh your browser.
Note: The final time that you refresh the resource and then select the virtual image, you receive an unable to load resource error message because the virtual image has been deleted at this point. After this, the view is refreshed.
6.36. SSL exception
In the command-line interface on Linux, you receive an SSL exception error.
Symptoms
On Linux, in the command-line interface of the Image Construction and Composition Tool, you receive the SSL exception error socket.sslerror: (-1, 'SSL exception').
Causes
If you are using the command-line interface on Linux and receive an SSL exception error, there might be a compatibility issue with the Java Runtime Environment (JRE) that you are using.
Resolving the problem
Use the java -version command to verify which JRE version you are using.
To resolve the communication error, you can use a different JRE. You can download IBM JRE version 6 for Linux from the following web address: http://www.ibm.com/developerworks/java/jdk/linux/download.html.
Once you have the JRE installed, you can set environment variables to allow this new JRE to be used by the command-line interface instead of the existing JRE. For example:
export JAVA_HOME=/opt/IBM/icct/jre export PATH=$JAVA_HOME/bin:$PATH
6.37. Cannot pass characters for arguments on the command line
You might not be able to pass certain characters for arguments on the command-line interface.
Symptoms
When invoking the command-line interface on Windows and passing arguments that contain an exclamation point (!), this character is removed from the argument during processing.
Causes
The command-line interface is based on Python. In certain cases, specific characters are used to indicate special characteristics.
Resolving the problem
When you want to pass the exclamation point as a command-line argument, use the caret character (^) to escape the exclamation point character. Also, enclose the argument in double quotation marks. For example, instead of "password!", use "password^!"
6.38. TCP/UDP firewall ports might be appended incorrectly on Red Hat system after synchronization
The TCP/UDP firewall ports might not be appended correctly on a Red Hat Linux system after synchronization.
Symptoms
If a software bundle has firewall rules defined when the bundle was added to a Red Hat image through the VMware ESX cloud provider, the TCP/UDP firewall ports might be appended incorrectly after synchronization.
Resolving the problem
Perform the following steps:
- Run: service iptables save
- Run: service iptables stop
- Run: cd /etc/sysconfig. Open the iptables file. If the new firewall rules are appended after the reject rules, move the new firewall rules before reject rules. If the new firewall rules are UDP port instead of TCP port, change tcp to udp for the new firewall rule.
- Run: service iptables start
6.39. Virtual image synchronization problems
Determine the reason why the virtual image synchronization fails.
6.39.1. Virtual image synchronization takes a long time
Your virtual image synchronization process takes a long time to complete.
Symptoms
Virtual image synchronization takes a long time and might time out and fail.
Causes
The cloud performance is poor. Cloud performance is not always consistent and can vary depending on usage and your connection. Depending on the cloud performance at any particular time, the virtual image synchronization can take longer to complete. Image Construction and Composition Tool continues the synchronization process in case there are performance issues. If the synchronization process is very slow and the performance is particularly poor, the synchronization eventually times out and you are notified that the synchronization failed because of performance issues.
Resolving the problem
After the virtual image synchronization process has timed out, you can retry the synchronization. If you retry and are still having synchronization issues, check the log files to determine the particular cloud provider issue that is affecting synchronization.
6.39.2. Virtual image synchronization fails after five hours for images
Virtual image synchronization fails after five hours for images on the IBM PureApplication System and IBM SmartCloud Provisioning cloud provider.
Symptoms
Virtual image synchronization fails after five hours even if the image deployment is still running on IBM PureApplication System.
Causes
By default, the Image Construction and Composition Tool waits up to five hours for the deployment of an image on IBM PureApplication System or IBM SmartCloud Provisioning. If the deployment does not complete within that time, the Image Construction and Composition Tool returns a failure.
Resolving the problem
In the configuration.config file in the <ICCT_INST_DIR>/icn.app/config/ directory, edit the parameter maxIWDDeployWaitTime to increase the wait time. The default value is 300 minutes which is five hours.
Restart the Image Construction and Composition Tool to apply the changes.
6.39.3. Synchronization fails for a virtual image imported from a running virtual machine
If you import a virtual image from a running virtual machine, the virtual image does not synchronize if Network Manager is running on the virtual machine. Network Manager must be disabled on the virtual machine.
Symptoms
The virtual image does not synchronize.
Causes
Network Manager has not been disabled on the virtual machine.
Resolving the problem
Disable Network Manager on the virtual machine and import the virtual image again.
To disable network manager, complete a procedure similar to the following:
- From the command line, run the following:
service NetworkManager stop- Run the following command:
chkconfig NetworkManager off- Ensure that the interfaces are not controlled by Network Manager by editing the following items in the /etc/sysconfig/network-scripts/ifcfg-eth0 file:
NM_controlled = no and ONBOOT=yesAfter you have disabled Network Manager on the virtual machine, try to import and synchronize the virtual image again.
There are many reasons for a synchronization failure. To help determine the problem, check the Image Construction and Composition Tool log files to debug.
6.39.4. Virtual image synchronization when using a trial version of VMware ESX
If you attempt to synchronize an image using a trial version of VMware ESX, the synchronization fails when attempting to copy the disk images to the VMware data store.
Symptoms
Image synchronization fails and the log contains messages similar to the following example:
Caused by: (0)null at org.apache.axis.transport.http.HTTPSender.readFromSocket(HTTPSender.java:744) at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)orjava.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: com.ibm.venture.vm.VirtualizationShimException: Remote exception. No details available. shim=https://localhost/sdk_copy26 endpoint=https://localhost/sdk_copy26 localizedMessage="Remote exception. No details available." causeType=org.apache.axis.AxisFault cause=(0)nullCauses
The VMware ESX server is a trial version. Only licensed VMware ESX servers are supported.
Resolving the problem
Use a licensed version of VMware ESX for your VMware ESX cloud provider.
6.39.5. Virtual image synchronization fails because of return code value
Virtual image synchronization fails if you use a script for the bundle installation that produces a return code value that is greater than zero. The Image Construction and Composition Tool checks the return value of the installation scripts used in bundles. A zero return code indicates success and the synchronization process can proceed.
Symptoms
Virtual image synchronization fails with the error: Unable to initiate add of execution bundle to VM.
Causes
The script used to install software in a bundle does not return a zero value.
Resolving the problem
Make sure that the scripts that you are using to install the software return a zero value to indicate successful installation. A value greater than zero indicates a failure and the virtual image cannot be synchronized. The installation script checks for error conditions relevant to the installation of the bundle. For errors that might compromise the bundle installation, an exit code greater than zero must be returned, indicating failure. If a script returns no exit code explicitly, the exit code of the last command of the script is implicitly returned. Make sure that the bundle installation script can handle parameters passed in the following format:
scriptname -parameter1 <parameter1 value> -parameter2 <parameter2 value>A sample installation script is as follows:#!/bin/sh # # Licensed Materials - Property of IBM # (C) Copyright IBM Corp. 2011 # All Rights Reserved # US Government Users Restricted Rights -Use, duplication or # disclosure restricted by GSA ADP Schedule Contract with IBM Corp. # # ---------------------------------------------------------------- # # Description: # Sample install script for an ICON bundle # # function usage () { echo "" echo " Usage :" echo " installBundleSample.sh -parm1 \"parm1value\" -parm2 \"parm2value\"" echo "" exit 1 } echo "$0: === Starting Install of Bundle Sample === " echo echo "$0: Arguments are: $* " echo #Init MY_PARM1="Default parm1 value" MY_PARM2="Default parm2 value" echo "$0: Parsing Arguments " echo #Read arguments while [ $# -ne 0 ] do case $1 in -parm1*) MY_PARM1=$2 ;; -parm2*) MY_PARM2=$2 ;; *) ;; esac shift 1 done # Now perform the work and actually do the install stuff - check the return code echo "$0: About to do the real work..." echo parm1 is "$MY_PARM1" parm2 is "$MY_PARM2" if [ $? -gt 0 ]; then echo "$0: echo $MY_PARM1 $MY_PARM2 failed" >&2 exit 1 fi # finish up and return zero to indicate everything went ok # (non-zero return code signifies an error) echo " " echo "$0: Done." exit 0
6.39.6. Virtual image synchronization fails and troubleshooting is not possible
If the synchronization fails, it is difficult to troubleshoot the problem because the deployed image and the bundles used for synchronization are removed.
Symptoms
The synchronization fails and it is difficult to perform problem determination because the deployed image and the compressed package are removed. Problems with the bundle installation scripts are difficult to troubleshoot.
Resolving the problem
Use the ICCT_DEBUG variable to troubleshoot the problem.
If this variable is not defined on your system, you can set it up by following these steps:
- If Image Construction and Composition Tool is started, stop it:
cd <install_dir> ./stop.sh- Define the ICCT_DEBUG variable:
export ICCT_DEBUG=true- Restart Image Construction and Composition Tool:
./start.sh
If later you no longer need the ICCT_DEBUG variable, you can disable it:
cd <install_dir> ./stop.sh export ICCT_DEBUG= ./start.sh
6.39.7. Windows image synchronization fails
Synchronization of a Windows image fails.
Symptoms
Windows image synchronization to the virtual machine can fail if the /etc/hosts file, required by the RXA protocol, in the Image Construction and Composition Tool server configuration is not configured correctly.
Resolving the problem
In addition to the localhost entry, the machine where the Image Construction and Composition Tool is installed must contain an entry with the Image Construction and Composition Tool server IP and the related host name and fully qualified host name, for example:
9.168.12.202 nc12202 nc12202.it.ibm.com
6.39.8. Cannot synchronize Windows 2008 R2 image
If an image is extended from another image, which is imported from a Windows 2008 R2 virtual machine running, the image synchronization might fail.
Symptoms
The synchronization fails and the image status is Virtural machine failed to start. If you open the VMware vCenter console, a window is displayed with the error
autochk program not found - skipping AUTOCHECKIf the synchronization fails, Image Construction and Composition Tool rolls back the changes, and the deployed virtual machine becomes Orphaned as a part of the rollback. The error window disappears.Resolving the problem
This problem is resolved in VMware ESX 4.1 Update 2 and VMware ESX 5.0 Update 1. For more information, see the following article in the VMware Knowledge Base: Cloned Windows 2008 R2 virtual machine fails to boot with the error: autochk program not found - skipping AUTOCHECK.
6.39.9. Timeout with Linux bundle in Windows image on VMware
Timeout with no error when synchronizing a Linux bundle inside a Windows image on a VMware cloud provider
Symptoms
Synchronizing a Linux bundle inside a Windows image on a VMware cloud provider times out with no error.
Causes
If the installation script used in a software bundle is not recognized by Windows as a known executable, the installation process executed during synchronization runs without stopping. The synchronization task fails after the timeout.
Resolving the problem
The timeout expires after six hours and the synchronization fails. At this point, you can remove the incorrect Linux bundle and add the correct one.
6.39.10. Image synchronization from the command line does not work
If you attempt to use the command line (CLI) to synchronize an image, the operation fails.
Symptoms
When you are using the CLI to synchronize a virtual image with an IBM PureApplication System or IBM SmartCloud Provisioning cloud provider, the image synchronization fails.
Resolving the problem
- To get a list of all possible parameters, run the following command:
allCloudSyncParams = image.getSynchronizeParametersFromCloudProvider()This command returns a list similar to the following example:[ { "label": "Select an environment profile for deployment:", "validValues": [ { "label": "envProfile10 (PureSystems_ESX)", "validValues": [ { "label": "CloudGroupLarge", "validValues": [ { "label": "ipgroup10", "value": "1013" } ], "isPassword": "False", "description": "cloud group", "type": "string", "value": "12", "name": "ipGroup" } ], "isPassword": "False", "description": "environment profile", "type": "string", "value": "10", "name": "cloudGroup" }, { "label": "none", "validValues": [ { "label": "CloudGroupLarge (PureSystems_ESX)", "validValues": [], "isPassword": "False", "description": "cloud group", "type": "string", "value": "12", "name": "ipGroup" } ], "isPassword": "False", "description": "environment profile", "type": "string", "value": "0", "name": "cloudGroup" } ], "isPassword": "False", "description": "select an environment profile.", "type": "profile", "name": "profileList" } ]- Based on the retrieved list, build a new JSON object with a structure similar to the following example:
myCloudSyncParams = [ {'name':'profileList', 'type':'string', 'value':''}, {'name':'envprofile', 'type':'string', 'value':'10'}, {'name':'cloudgroup', 'type':'string', 'value':'12'}, {'name':'ipgroup', 'type':'string', 'value':'1013'}, {'name':'username', 'type':'string', 'value':'root'}, {'name':'password', 'type':'string', 'value':'passw0rd'} ]In this JSON object:
- The profileList element has an empty value.
- You must add the username and password elements to the JSON object. They are not returned by the getSynchronizeParametersFromCloudProvider() method.
For assistance in building this JSON object, use the command line with the -f parameter and sample file sample/createSynchronizeParametersForCloudProvider.py.
- To get the list of possible image parameters, run the following command:
myImgSyncParams = getSynchronizeParametersFromImage()- To synchronize the image, run the following command:
image.synchronize(myImgSyncParams, myCloudSyncParams, True)
For assistance in building this JSON object, use the command line with the -f parameter and sample file sample/createSynchronizeParametersForCloudProvider.py.
Troubleshoot Advanced Middleware Configuration
Discover solutions or workarounds for common problems.
7.1. Troubleshoot the generation process
If the generation process does not create the required objects on the framework server, try these common solutions.
- Make sure you have met all the prerequisites
- Make sure generation completed successfully
- Check the output of the generation process for error messages.
- Open the RAFW_HOME/user/preserve/envgen/env-config.properties file and make sure you can read it.
- Check the services log
- If the use Tomcat as your application server, the default log directory is /opt/IBM/RAFServer/server/tomcat/logs.
- If you use WebSphere Application Server, the information is sent to the SystemOut.log file in the logs directory of your server.
- Verify your permissions
- Make sure the application server user (either the Tomcat user or the WebSphere Application Server user) has write access to the work, user, and log directories.
- Make sure you can SSH into the target system using the target operating system user (OS_USER) credentials from the configuration definition.
- Make sure the target operating system user has permission to write to the OS_RAFW_HOME directory on the target system.
7.2. Clearing Jython cache
If the Jython cache files on a remote system are out of date, clear the Jython cache by running the rafw_delete_jython_class_files action at the node scope of the remote system. The error message that indicates that the Jython cache is out of date is shown below. In particular, the ImportError and NameError lines in the error message indicate the problem.
call_wsadmin_function: [exec] WASX7209I: Connected to process "dmgr" on node dmgr using SOAP connector; The type of process is: DeploymentManager [exec] WASX7017E: Exception received while running file "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/jythonLib.py"; exception information: com.ibm.bsf.BSFException: exception from Jython: [exec] Traceback (innermost last): [exec] File "<string>", line 71, in ? [exec] File "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/BFWCommand.py", line 18, in ? [exec] File "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/ConfigMediator.py", line 17, in ? [exec] File "/opt/IBM/RAFServer/rafw/product/lib/jython/wsadmin_lib/ConfigComparor.py", line 16, in ? [exec] ImportError: No module named bfw [exec] [exec] WASX7015E: Exception running command: "BFWCommand.execute('-scope cell -scopename qa -nodename dmgr -properties /opt/IBM/RAFServer/rafw/user/environments/cycle/cells/qa/jms.xml -mode import -marker JMSProviders', ['JMSProvider'])"; exception information: [exec] com.ibm.bsf.BSFException: exception from Jython: [exec] Traceback (innermost last): [exec] File "<input>", line 1, in ? [exec] NameError: BFWCommand [exec]
Procedure
At the node scope of the target system, run the rafw_delete_jython_class_files action by entering the following command: ./rafw.sh -e env_name -c cell_name -n node_name -t rafw_delete_jython_class_files.
7.3. Test server connections
To verify that the network setup is completed correctly, test the connection between the framework server and the target computers. The framework server and the target computers can be UNIX, Linux, or Windows systems.
Target systems on UNIX communicate with the framework server by using the SSH protocol with either username and password or SSH key authentication. Keys can be managed using the standard operating system utilities. your framework server is a Windows system and the target computer is a UNIX or Linux system, you must use third party software to test SSH connections.
Procedure
To test the framework server and target system connection:
- When the framework server and the target computer are UNIX or Linux systems, and you are using SSH keys:
- On the framework server, enter the following command:
[rafwadm RAFW_HOME]$ ssh -i user/keys/ssh_id OS_USER@HOST_NAME
where OS_USER is the name of the operating system user, HOST_NAME is the name of the target system host machine, and ssh_id is the SSH key ID.
- When prompted, enter the password that you used when you created the SSH key.
- When the framework server and the target computer are UNIX or Linux systems and you are not using SSH keys:
- On the framework server, enter the following command:
[rafwadm RAFW_HOME]$ ssh OS_USER@HOST_NAME
where OS_USER is the name of the operating system user and HOST_NAME is the name of the target system host machine.
- When prompted, enter the password of the OS_USER.
- When the framework server is a UNIX or Linux system, and the target system is a Windows system:
- On the framework server, enter the following command:
smbclient //targethost/c$
where targethost is the network address of the target system host computer.
- When the framework server and the target computer are Windows systems:
- On the framework server, enter the following command:
net use z: \\targethost\c$
where targethost is the network address of the target system host computer.
- When the framework server is a Windows system, and the target system is a UNIX or Linux system:
- Use a third party software to create an external SSH connection. Then test the connection in the same way as shown in the step When the framework server and the target computer are UNIX or Linux systems.
7.4. Connection to UNIX or Linux systems fails with 'Broken pipe' message
If interactive commands exist in the user profile, a secure shell (SSH) connection to remote UNIX or Linux systems fails with a broken pipe message in the Remote Execution and Access (RXA) log file. To troubleshoot this error, you must remove the interactive commands from the user shell profile.
Diagnosing the error
If connection to a remote UNIX or Linux system fails, determine if the error is due to interactive commands in the user profile by performing the following steps:
- Enable RXA tracing.
- From the framework server, run an action that requires a connection to the target UNIX or Linux system.
- In the RXA trace output, look for the following lines: java.io.IOException: Read end dead and Broken pipe.
- On the framework server, enter the following command: ssh remoteuser@remote.server.name uname. If prompted, enter the password.
- If no interactive commands exist in the user profile, the command returns only the output of the uname command, for example, Linux.
- If interactive commands exist in the user profile, the command returns additional data in the output, such as stty: statndard input: Invalid argument.
Troubleshoot the error
If the connection error is due to interactive commands in the user profile, remove the interactive commands from the user profile on the remote system. Based on the shell specified for the remote user, the files to edit might be different. For example, for users of the bash shell, the files to edit might be /etc/profile ~/.bash_profile and /etc/profile ~/.bashrc.Verify that the interactive commands are removed by entering the command in step 4 above. The command output must be the value of uname only.
7.5. Troubleshoot installation errors
If your installation actions fail or return errors, you might want to try some common troubleshooting techniques.
- Review the installation prerequisites
- Ensure that you meet the installation prerequisites, such as the software and hardware requirements, to install the product.
- Review the installation response files
- If your installation actions fail, locate and read the installation response file for the product. The installation response file supplied by the product contains information about the parameters required for the installation. Installation failure or errors might be due to missing or incorrectly-supplied installation parameters.
7.6. Troubleshoot Workload Deployer errors
Troubleshoot errors with PureApplication System processes by referring to the described problems and the possible resolutions. If the problem cannot be resolved, you must restart the process.
List of errors
Process Problem Resolution Connect to server Script Package log shows the error: "Could not send Message."
- Verify the host, domain, username, password, and SSL options.
- Verify that the Advanced Middleware Configuration for PureApplication System server is available by logging in to the web client.
- Retry the pattern deployment.
Create a new Advanced Middleware Configuration configuration environment Configuration environment and projects not created Check the logs for details:
- Check the services log at /opt/IBM/RAFServer/server/tomcat/logs/raf.2012-03-16.log
- Check the server log at /opt/IBM/RAFServer/server/tomcat/logs/catalina.out
- Check the framework server log at /opt/IBM/RAFServer/server/tomcat/logs/framework.log.0
- Create a new Advanced Middleware Configuration configuration environment
- Update the Advanced Middleware Configuration environment pattern to match a newly supplied pattern.
Advanced Middleware Configuration server is unable to connect to the newly deployed target system. Check product logs and enable tracing:
- Check the framework server log at RAFW_HOME/logs/framework.log.0
- Check the action log at RAFW_HOME/logs/scope/action_name.trace
- Enable RXA tracing and try connecting to the server again.
Fix errors, if any, in the job:
- Check job results in the Advanced Middleware Configuration web client and fix any errors.
- Restart the job and verify that it completes successfully.
- Retry the pattern deployment.
Permission denied to create working directory on the target host.
- Edit the value for RAFW_HOME_PATH in the script package settings to match the value for LINUX_RAFW_HOME in the file RAFW_HOME/configure.properties.
- Retry the pattern deployment.
Find the automation plan name for a Workload Deployer cell The Advanced Middleware Configuration environment was not created (see previous row).
- Verify that the RAFW_env_cell_execute or RAFW_env_cell_import automation plan exists on the Advanced Middleware Configuration server
- Verify that the user defined as the RAFW_USER has access to view the environment group.
- Retry the pattern deployment.
Launch the project Errors when the project runs.
- Check job results in the web client and fix any errors.
- Restart the job and verify that it completes successfully.
You do not need to retry the pattern deployment because this is the last step in the script package.
7.7. Missing configuration in Advanced Middleware Configuration console after upgrade
After you upgrade to Advanced Middleware Configuration version 1.1, an existing configuration definition might not be visible in the Advanced Middleware Configuration console.
The configuration properties file is named env-config.properties. In versions of Advanced Middleware Configuration previous to 1.1, the configuration properties file was in the RAFW_HOME/work directory. Beginning with version 1.1, the configuration properties file is in the RAFW_HOME/user/preserve/envgen/ directory. The console searches the envgen directory when displaying available configurations on the Configuration page. Configuration properties files created in the work directory might not be displayed.
- If you can see the missing file in theRAFW_HOME/work directory, copy the file to the RAFW_HOME/user/preserve/envgen/ directory.
- If you cannot see the missing file in the RAFW_HOME/work directory, but the configuration definition exists in the RAFW_HOME/user/environments directory, you can recreate the configuration properties file. Run the existing configuration option of the configuration editor and point it to the live configuration that existing configuration definition represents. You do not need to generate the configuration definition after you finish editing the configuration properties file.
7.8. Automation plan fails with no output in results
If your automation plan fails, but the Results display contains no messages, there might be a problem with the RAFW_LOCAL server for the automation engine.
- Start the automation web client by opening a browser and specifying the host name for your framework server followed by a slash (/) and the letters "bfui", as in the following example:
framework.server.example.com/bfui- On the Console page, go to the menu on the left side of the page and click Servers. The RAFW_LOCAL server should be displayed with a green icon to indicate that it is running.
- If the server is not running, click the server name. The server information is displayed in the lower pane.
- Click Test Connection. A server test is run and the results are displayed on the Test Results page.
- If you see a Status: Authentication failure message, complete the following steps:
- Click Server Auth on the left side of the page.
- Click RAFW_LOCAL to display the server authorization information for the server.
- Enter the correct password for the displayed user in response to the Password and Verified prompts.
- Click Servers again.
This problem is commonly caused by someone changing the Server Auth user password, but not updating the password in the web client.
- If you see a Status: Connection failure message, complete the following steps:
- Ensure that the Host name or IP address shown on the Details page is correct.
- Ping the server and make sure it is responding.
- Verify that the automation agent is running:
- On Linux, enter the following command:
ps -ef | grep bfagent- On Windows, check the Task Manager for the bfagent task.
If the agent is not running, browse to the Rational Build Forge Information Center and search for "Troubleshooting agents."
7.9. Garbled data in .properties file
Java .properties files are encoded by default as ISO-8859-1. Advanced Middleware Configuration requires that .properties files be in UTF-8 encoding.
Advanced Middleware Configuration recognizes standard .properties files, such as the following files, at any scope, and overrides the default Java encoding:
- configure.properties
- promote.properties
- scope.properties
The problem occurs when you create a new .properties file with a name that is not recognized. For example, you might copy a promote.properties file to a template and rename the file my_data.properties.
The solution is to manually change the file encoding by completing the following steps:
- In the Eclipse client, right-click the file name and click Properties.
- Under Text file encoding, click Other and then select UTF-8.
- Click OK to close the window.
- You might see a warning message saying the new encoding conflicts with the content type. Click Yes to continue with the change.
8.1. Error messages overview
Most of the Rational error messages begin with a code such as CRRE and end with a message type such as E or I. Use the prefix to isolate the probable cause or location of the error and the message type to determine the severity of the error. Any other kind of error message relates to the databases, application servers, and other components used by the Rational product. Sometimes, the messages have an RSS feed that you can subscribe to. This topic explains the anatomy of the error messages.
Error message prefixes
Use the message prefix to identify the component that caused the error. These prefixes are specific to Rational Automation Framework:
- CRWBP
- IBM WebSphere Enterprise Service Bus, IBM WebSphere Process Server , IBM WebSphere Service Registry and Repository
- CRWEB
- Automation web client
- CRWFW
- Automation framework
- CRWIH
- IBM HTTP Server
- CRWSC
- Automation services client
- CRWSE
- Automation services server
- CRWVE
- IBM WebSphere Virtual Enterprise
- CRWWA
- IBM WebSphere Application Server
- CRWWP
- IBM WebSphere Portal
Error message suffixes
Use the message suffix to determine the severity of the error. These suffixes are used in the error messages:
- A
- Audit message, indicating the results of an audit.
- C
- Configuration message, indicating the current product configuration. This message does not indicate configuration being managed by the product in other systems, but instead displays the configuration of the product.
- E
- Error message, alerting you to a problem that must be solved before you can continue with your task.
- I
- Informational message, describing status or providing information for normal conditions and operations.
- W
- Warning message, alerting you to a condition that might cause problems in the future. You can generally continue with your tasks, but those tasks might not complete in a way that is expected.
Sometimes, even though the error message has a code similar to CRW, the error might be caused by a database, application server, or another product used with Rational Automation Framework. To accurately isolate the cause of the error, see the following log files:
- Automation engine logs
- Database logs
- Logs for the automation target
For specific log file locations, check the product documentation for the appropriate product. For example, you can find WebSphere Application Server logs in the WAS_PROFILE_HOME directory.
RSS feed
If the message has an RSS feed and if you subscribe to the feed, you are notified every time that a technical document relating to the error goes live on World Wide Web. For more information about RSS feeds, visit the "Learn more about RSS feeds" web page at http://www.ibm.com/ibm/syndication/us/en/howto.html.