Home

Troubleshoot WebSphere eXtreme Scale

+

Search Tips   |   Advanced Search

  1. Troubleshooting and support for WebSphere eXtreme Scale
  2. Enable logging
  3. Collecting trace
  4. Troubleshooting with High Performance Extensible Logging (HPEL)
  5. Analyzing log and trace data
  6. Troubleshooting the product installation
  7. Troubleshooting client connectivity
  8. Troubleshooting cache integration
  9. Troubleshooting the JPA cache plug-in
  10. Troubleshooting IBM eXtremeMemory
  11. Troubleshooting administration
  12. Troubleshooting data monitoring
  13. Troubleshooting multiple data center configurations
  14. Troubleshooting loaders
  15. Troubleshooting XML
  16. Troubleshooting deadlocks
  17. Troubleshooting lock timeout exceptions for a multi-partition transaction
  18. Troubleshooting security
  19. Troubleshooting Liberty profile configurations
  20. Collecting data with the IBM Support Assistant Data Collector
  21. IBM Support Assistant for WebSphere eXtreme Scale


Troubleshooting and support for WebSphere eXtreme Scale

To isolate and resolve problems with your IBM products, we can use the troubleshooting and support information. This information contains instructions for using the problem-determination resources that are provided with your IBM products, including WebSphere eXtreme Scale .


Techniques for troubleshooting problems

Troubleshooting is a systematic approach to solving a problem. The goal of troubleshooting is to determine why something does not work as expected and how to resolve the problem. Certain common techniques can help with the task of troubleshooting.

The first step in the troubleshooting process is to describe the problem completely. Problem descriptions help you and the IBM technical-support representative know where to start to find the cause of the problem. This step includes asking yourself basic questions:

The answers to these questions typically lead to a good description of the problem, which can then lead you to a problem resolution.


What are the symptoms of the problem?

When starting to describe a problem, the most obvious question is "What is the problem?" This question might seem straightforward; however, we can break it down into several more-focused questions that create a more descriptive picture of the problem. These questions can include:


Where does the problem occur?

Determining where the problem originates is not always easy, but it is one of the most important steps in resolving a problem. Many layers of technology can exist between the reporting and failing components. Networks, the data grid, and servers are only a few of the components to consider when we are investigating problems.

The following questions help you to focus on where the problem occurs to isolate the problem layer:

If one layer reports the problem, the problem does not necessarily originate in that layer. Part of identifying where a problem originates is understanding the environment in which it exists. Take some time to completely describe the problem environment, including the operating system and version, all corresponding software and versions, and hardware information. Confirm that we are running within an environment that is a supported configuration; many problems can be traced back to incompatible levels of software that are not intended to run together or have not been fully tested together.


When does the problem occur?

Develop a detailed timeline of events leading up to a failure, especially for those cases that are one-time occurrences. We can most easily develop a timeline by working backward: Start at the time an error was reported (as precisely as possible, even down to the millisecond), and work backward through the available logs and information. Typically, you need to look only as far as the first suspicious event that you find in a diagnostic log.

To develop a detailed timeline of events, answer these questions:

Responding to these types of questions can give you a frame of reference in which to investigate the problem.


Under which conditions does the problem occur?

Knowing which systems and applications are running at the time that a problem occurs is an important part of troubleshooting. These questions about your environment can help you to identify the root cause of the problem:

Answering these types of questions can help you explain the environment in which the problem occurs and correlate any dependencies. Remember that just because multiple problems might have occurred around the same time, the problems are not necessarily related.


Can the problem be reproduced?

From a troubleshooting standpoint, the ideal problem is one that can be reproduced. Typically, when a problem can be reproduced you have a larger set of tools or procedures at your disposal to help you investigate. Consequently, problems that we can reproduce are often easier to debug and solve.

However, problems that we can reproduce can have a disadvantage: If the problem is of significant business impact, you do not want it to recur. If possible, recreate the problem in a test or development environment, which typically offers you more flexibility and control during your investigation.


Searching knowledge bases

We can often find solutions to problems by searching IBM knowledge bases. We can optimize your results by using available resources, support tools, and search methods.

We can find useful information by searching the information center for WebSphere eXtreme Scale . However, sometimes you need to look beyond the information center to answer your questions or resolve problems.

To search knowledge bases for information that you need, use one or more of the following approaches:


Getting fixes

A product fix might be available to resolve your problem.

To find and install fixes:

  1. Obtain the tools required to get the fix. Use the IBM Update Installer to install and apply various types of maintenance packages for WebSphere eXtreme Scale or WebSphere eXtreme Scale Client. Because the Update Installer undergoes regular maintenance, use the most current version of the tool.

  2. Determine which fix you need. Recommended fixes for WebSphere eXtreme Scale to select the latest fix. When you select a fix, the download document for that fix opens.

  3. Download the fix. In the download document, click the link for the latest fix in the "Download package" section.

  4. Apply the fix. Follow the instructions in the "Installation Instructions" section of the download document.

  5. Subscribe to receive weekly e-mail notifications about fixes and other IBM Support information.


Getting fixes from Fix Central

Use Fix Central to find the fixes that are recommended by IBM Support for a variety of products, including WebSphere eXtreme Scale . With Fix Central, we can search, select, order, and download fixes for your system with a choice of delivery options. A WebSphere eXtreme Scale product fix might be available to resolve your problem.

To find and install fixes:

  1. Obtain the tools that are required to get the fix. If it is not installed, obtain your product update installer. We can download the installer from Fix Central . This site provides download, installation, and configuration instructions for the update installer.

  2. Select as the product, and select one or more check boxes that are relevant to the problem to resolve.

  3. Identify and select the fix that is required.

  4. Download the fix.

    1. Open the download document and follow the link in the "Download Package" section.
    2. When downloading the file, ensure that the name of the maintenance file is not changed. This change might be intentional, or it might be an inadvertent change that is caused by certain web browsers or download utilities.

  5. Apply the fix.

  6. Optional: Subscribe to receive weekly e-mail notifications about fixes and other IBM Support updates.


Contacting IBM Support

IBM Support provides assistance with product defects, answers FAQs, and helps users resolve problems with the product.

After trying to find your answer or solution by using other self-help options, such as release notes, we can contact IBM Support. Before contacting IBM Support, your company or organization must have an active IBM maintenance contract, and you must be authorized to submit problems to IBM. For information about the types of available support, see the Support portfolio topic in the "Software Support Handbook".

To contact IBM Support about a problem:

  1. Define the problem, gather background information, and determine the severity of the problem. For more information, see the Getting IBM support topic in the Software Support Handbook.

  2. Gather diagnostic information.

  3. Submit the problem to IBM Support in one of the following ways:
    • With IBM Support Assistant (ISA).
    • Online through the IBM Support Portal : We can open, update, and view all of your service requests from the Service Request portlet on the Service Request page.
    • By phone: For the phone number to call in your region, see the Directory of worldwide contacts web page.


Results

If the problem that you submit is for a software defect or for missing or inaccurate documentation, IBM Support creates an Authorized Program Analysis Report (APAR). The APAR describes the problem in detail. Whenever possible, IBM Support provides a workaround that we can implement until the APAR is resolved and a fix is delivered. IBM publishes resolved APARs on the IBM Support website daily, so that other users who experience the same problem can benefit from the same resolution.


Exchanging information with IBM

To diagnose or identify a problem, you might need to provide IBM Support with data and information from your system. In other cases, IBM Support might provide you with tools or utilities to use for problem determination.


Sending information to IBM Support

To reduce the time that is required to resolve your problem, we can send trace and diagnostic information to IBM Support.

Procedure

To submit diagnostic information to IBM Support:

  1. Open a problem management record (PMR).

  2. Collect the diagnostic data that you need. Diagnostic data helps reduce the time that it takes to resolve your PMR. We can collect the diagnostic data manually or automatically:

    • Collect the data manually.
    • Collect the data automatically.

  3. Compress the files using the .zip or .tar file format.

  4. Transfer the files to IBM. Use one of the following methods to transfer the files to IBM:

    If we are using a z/OS product and you use ServiceLink / IBMLink to submit PMRs, we can send diagnostic data to IBM Support in an e-mail or by using FTP.

    All of these data exchange methods are explained on the IBM Support website .


Receiving information from IBM Support

Occasionally an IBM technical-support representative might ask you to download diagnostic tools or other files. Use FTP to download these files.

Before you begin

Ensure that your IBM technical-support representative provided you with the preferred server to use for downloading the files and the exact directory and file names to access.


Procedure

To download files from IBM Support:

  1. Use FTP to connect to the site that your IBM technical-support representative provided and log in as

    anonymous. Use your e-mail address as the password.

  2. Change to the appropriate directory:

    1. Change to the /fromibm directory. 
      cd fromibm

    2. Change to the directory that your IBM technical-support representative provided. 
      cd nameofdirectory

  3. Enable binary mode for the session. 
    binary

  4. Use the get command to download the file that your IBM technical-support representative specified. 
    get filename.extension

  5. End your FTP session. 
    quit


Subscribing to Support updates

To stay informed of important information about the IBM products that you use, we can subscribe to updates.

By subscribing to receive updates about the product, we can receive important technical information and updates for specific IBM Support tools and resources. We can subscribe to updates by using one of two approaches:

Social media subscriptions

The following RSS feed is available for the product:

For general information about RSS, including steps for getting started and a list of RSS-enabled IBM web pages, visit the IBM Software Support RSS feeds site.

My Notifications

With My Notifications, we can subscribe to Support updates for any IBM product. My Notifications replaces My Support, which is a similar tool that you might have used in the past. With My Notifications, we can specify to receive daily or weekly e-mail announcements. We can specify what type of information we want to receive, such as publications, hints and tips, product flashes (also known as alerts), downloads, and drivers. My Notifications enables you to customize and categorize the products about which we want to be informed and the delivery methods that best suit your needs.

To subscribe to Support updates:

  1. Subscribe to the RSS feed for the WebSphere eXtreme Scale forum .

    1. On the subscription page, click the RSS feed icon.

    2. Select the option to use to subscribe to the feed.

    3. Click Subscribe.

  2. Subscribe to My Notifications by going to the IBM Support Portal and click My Notifications in the Notifications portlet.

  3. Sign in using your IBM ID and password, and click Submit.

  4. Identify what and how we want to receive updates.

    1. Click the Subscribe tab.

    2. Select the appropriate software brand or type of hardware.

    3. Select one or more products by name and click Continue.

    4. Select your preferences for how to receive updates, whether by e-mail, online in a designated folder, or as an RSS or Atom feed.

    5. Select the types of documentation updates to receive, for example, new information about product downloads and discussion group comments.
    6. Click Submit.


Results

Until you modify your RSS feeds and My Notifications preferences, you receive notifications of updates that you have requested. We can modify your preferences when needed; for example, if you stop using one product and begin using another product.

IBM Software Support RSS feeds
Subscribe to My Notifications support content updates
My Notifications for IBM technical support
My Notifications for IBM technical support overview


Enable logging

Use logs to monitor and troubleshoot your environment.

Logs are saved different locations and formats depending on the configuration.


What to do next

View the log files in their specified locations. Common messages to look for in the SystemOut.log file are start confirmation messages, such as the following example:

CWOBJ1001I: ObjectGrid Server catalogServer01 is ready to process requests.


Configure remote logging

We can enable remote logging to save log entries on a remote server. Remote logging can be helpful when set a detailed debugging log level to help isolate a problem or monitor behavior over a long time period.

Use remote logging for analysis of historical data. The servers in your environment keep a limited number of log files in the system. Configure remote logging if you require more log files to be saved for further analysis. The remote logging server aggregates the data from multiple servers. We can configure your entire topology of catalog servers and container servers to send files to the same remote logging server.

  1. Configure remote logging on each catalog server or container server. Enable remote logging by editing the following properties in the server properties file:

    syslogEnabled

    Enables remote logging for analysis of historical data. You must have a syslog server available to listen for and capture events.

    Default: false

    syslogHostName

    Host name or IP address of the remote server on which we want to log historical data.

    syslogHostPort

    Port number of the remote server on which we want to log historical data.

    Valid values: 0-65535

    Default: 512

    syslogFacility

    Indicates the type of remote logging facility that is being used.

    Valid values: kern, user, mail, daemon, auth, syslog, lpr, news, uucp, cron, authpriv, ftp, sys0, sys1, sys2, sys3,local0, local1, local2, local3, local4, local5, local6, local7

    Default:

    user

    syslogThreshold

    Specifies the threshold of the severity of messages to send to the remote logging server. To send both warning and severe messages, enter a value of WARNING. To send severe messages only, enter

    SEVERE.

    Valid values: SEVERE,
    WARNING

    Default:

    WARNING

  2. Restart the catalog servers and container servers on which you changed the properties.


Results

Messages are sent to your configured remote logging server for archival and analysis. .NET:

.NET client logs

Logs in a .NET client are configured by default and are written to files in the logs directory and the Windows event log.


Default log file locations and settings

After you install the WebSphere eXtreme Scale Client for .NET, the log directories are created, based on the log directory location that you specify during installation. The following log files are generated:


Trace and FFDC logs

Trace logs are not enabled by default on .NET clients. If you need to collect trace for a .NET client, contact the Support team for further assistance.


Collecting trace

Use trace to monitor and troubleshoot your environment.

You must provide trace for a server when you work with IBM support.

Collecting trace can help you monitor and fix problems in the deployment of WXS. How you collect trace depends on the configuration.


Results

Trace files are written to the specified location.


Server trace options

We can enable trace to provide information about your environment to IBM support.


About trace

WebSphere eXtreme Scale trace is divided into several different components. We can specify the level of trace to use for a catalog server or container server. Common levels of trace include: all, debug, entryExit, and event.

An example trace string follows: 

ObjectGridComponent=level=enabled
We can concatenate trace strings. Use the * (asterisk) symbol to specify a wildcard value, such as ObjectGrid*=all=enabled. If you need to provide a trace to IBM support, a specific trace string is requested. For example, if a problem with replication occurs, the ObjectGridReplication=debug=enabled trace string might be requested.


Trace specification

ObjectGrid

General core cache engine.

ObjectGridCatalogServer

General catalog service.

ObjectGridChannel

Static deployment topology communications.

ObjectGridClientInfo

DB2 client information.

ObjectGridClientInfoUser

DB2 user information.

ObjectgridCORBA

Dynamic deployment topology communications.

ObjectGridDataGrid

The AgentManager API.

ObjectGridDynaCache

The WebSphere eXtreme Scale dynamic cache provider.

ObjectGridEntityManager

The EntityManager API. Use with the Projector option.

ObjectGridEvictors

ObjectGrid built-in evictors .

ObjectGridJPA

JPA loaders.

ObjectGridJPACache

JPA cache plug-ins.

ObjectGridLocking

ObjectGrid cache entry lock manager.

ObjectGridLogHandler

Remote logging information.

ObjectGridMBean

Management beans.

ObjectGridMonitor

Historical monitoring infrastructure.

ObjectGridNative

WebSphere eXtreme Scale native code trace, including eXtremeMemory native code.

ObjectGridOSGi

The WebSphere eXtreme Scale OSGi integration components.

ObjectGridPlacement

Catalog server shard placement service.

ObjectGridQuery

ObjectGrid query.

ObjectGridReplication

Replication service.

ObjectGridRouting

Client/server routing details.

ObjectGridSecurity

Security trace.

ObjectGridSerializer

The DataSerializer plug-in infrastructure.

ObjectGridStats

ObjectGrid statistics.

ObjectGridTransactionManager

The WebSphere eXtreme Scale transaction manager.

ObjectGridWriteBehind

ObjectGrid write behind.

ObjectGridXA

Multi-partition transaction trace.

ObjectGridXM

General IBM eXtremeMemory trace.

ObjectGridXMEviction

eXtremeMemory eviction trace.

ObjectGridXMTransport

eXtremeMemory general transport trace.

ObjectGridXMTransportInbound

eXtremeMemory inbound specific transport trace.

ObjectGridXMTransportOutbound

eXtremeMemory outbound specific transport trace.

Projector

The engine within the EntityManager API.

QueryEngine

The query engine for the Object Query API and EntityManager Query API.

QueryEnginePlan

Query plan trace.

TCPChannel

The IBM eXtremeIO TCP/IP channel.

XsByteBuffer

WebSphere eXtreme Scale byte buffer trace.

Enabling logging

Collecting trace

Start stand-alone servers that use the ORB transport

Administering with the xscmd utility


Troubleshooting with High Performance Extensible Logging (HPEL)

HPEL is a log and trace facility that we can use in stand-alone and WebSphere Application Server environments. Use HPEL to store and access log, trace, System.err, and System.out information produced by the application server or applications. HPEL is an alternative to the basic log and trace facility, which provides the Java virtual machine (JVM) logs, diagnostic trace, and service log files. These files are commonly named SystemOut.log/SystemErr.log, trace.log and activity.log. HPEL provides a log data repository, a trace data repository, and a text log file.

Instead of the existing logging facility, we can use HPEL, which is disabled by default. In HPEL mode, the log and trace contents are written to a log data or trace data repository in a proprietary binary format. Therefore, disabling HPEL can improve server performance by providing faster log and trace handling capabilities. Enable HPEL with the server properties files for your container servers and catalog servers. After you enable HPEL, all WebSphere eXtreme Scale logging and the resulting log files are placed in the specified HPEL repository location.

  1. Set properties to enable HPEL logging. Edit the Server properties file for each container and catalog server with the properties to use.

    hpelEnable

    Specifies if High Performance Extensible Logging (HPEL) is enabled. HPEL logging is enabled when the property is set to true.

    Default: false

    hpelRepositoryLocation

    Specifies the HPEL logging repository location.

    Default: "." (the runtime location)

    hpelEnablePurgeBySize

    Indicates if the HPEL purges log files by size. We can set the size of the files with the hpelMaxRepositorySize property.

    Default: true (enabled)

    hpelEnablePurgeByTime

    Indicates if the HPEL purges log files by time. Set the amount of time with the hpelMaxRetentionTime property.

    Default: true (enabled)

    hpelEnableFileSwitch

    Indicates if the HPEL file is enabled to create a new file at a specified hour. Use the hpelFileSwitchHour property to specify the hour at which to create a new file.

    Default: false (disabled)

    hpelEnableBuffering

    Indicates if the HPEL buffering is enabled.

    Default: false (disabled)

    hpelIncludeTrace

    Indicates if the HPEL text files include tracing.

    Default: false (disabled)

    hpelOutOfSpaceAction

    Indicates the action to be performed when the disk space has been exceeded.

    Default:

    PurgeOld

    Possible values: PurgeOld, StopServer, StopLogging

    hpelOutputFormat

    Indicates the format of the log files to be generated.

    Default:

    Basic

    Possible values: Basic, Advanced, CBE-1.0.1

    hpelMaxRepositorySize

    Indicates the maximum size of files, in megabytes. This value is used when you able the hpelEnablePurgeBySize property.

    Default:

    50

    hpelMaxRetentionTime

    Indicates the maximum retention time to hold files, in hours.

    Default:

    48

    hpelFileSwitchHour

    Indicates the hour at which to create a new file. This value is used when the hpelEnableFileSwitch property is enabled.

    Default:

    0

  2. Restart the servers on which you modified the server properties file to set HPEL properties. After HPEL is enabled and the server restarted, the previous WebSphere eXtreme Scale logging information is no longer available. The previous logging information is replaced by equivalent HPEL information.

  3. Use the HPEL command-line log viewer to view your log files....

      cd /opt/IBM/WebSphere/eXtremeScale/ObjectGrid/bin
      ./logViewer.sh -help

    Create a legacy format log file, legacyFormat.log, that contains only log records INFO, WARNING, and SEVERE:

      ./logViewer.sh -outLog ../logs/legacyFormat.log -minLevel INFO -maxLevel SEVERE

    Use a text editor to view the legacy format log file that you created.

    View only the log records for thread 0:

      ./logViewer.sh -thread 0

    View only WARNING messages:

      ./logViewer.sh -level WARNING

    Retrieve all log records NOT from loggers that begin with com.ibm:

      ./logViewer.sh -excludeLoggers com.ibm.*

    Extract a repository of just WARNING and SEVERE messages and save the resulting file in a new directory:

      ./logViewer.sh -minLevel WARNING -maxLevel SEVERE -extractToNewRepository ../logs/newHPELRepository

    Export the contents of the resulting repository to a text format log file:

      ./logViewer.sh -repositoryDir ../logs/newHPELRepository -outLog ../logs/newFormat.log


Analyzing log and trace data

Use the log analysis tools to analyze how your runtime environment is performing and solve problems that occur in the environment.

We can generate reports from the existing log and trace files in the environment. These visual reports can be used for the following purposes:


Log analysis overview

Use the xsLogAnalyzer tool to help troubleshoot issues in the environment.


All failover messages

Displays the total number of failover messages as a chart over time. Also displays a list of the failover messages, including the servers that have been affected


All eXtreme Scale critical messages

Displays message IDs along with the associated explanations and user actions, which can save you the time from searching for messages.


All exceptions

Displays the top five exceptions, including the messages and how many times they occurred, and what servers were affected by the exception.


Topology summary

Displays a diagram of how your topology is configured according to the log files. Use this summary to compare to your actual configuration, possibly identifying configuration errors.


Topology consistency: Object Request Broker (ORB) comparison table

Displays ORB settings in the environment. Use this table to help determine if the settings are consistent across your environment.


Event timeline view

Displays a timeline diagram of different actions that have occurred on the data grid, including life cycle events, exceptions, critical messages, and first-failure data capture (FFDC) events.


Run log analysis

We can run the xsLogAnalyzer tool on a set of log and trace files from any computer.

  1. Enable logs and trace.

  2. Collect your log files.

    The log files can be in various locations depending on how you configured them. If we are using the default log settings, we can get the log files from the following locations:

    • In a stand-alone installation:

        wxs_install_root/bin/logs/<server_name>

    • In an installation that is integrated with WebSphere Application Server:

        was_root/logs/<server_name>

  3. Collect your trace files.

    The trace files can be in various locations depending on how you configured them. If we are using the default trace settings, we can get the trace files from the following locations:

    • In a stand-alone installation: If no specific trace value is set, the trace files are written to the same location as the system out log files.

    • In an installation that is integrated with WebSphere Application Server:

        was_root/profiles/server_name/logs

    Copy the log and trace files to the computer from which we are planning to use the log analyzer tool.

  4. To create custom scanners in your generated report, create a scanner specifications properties file and configuration file before we run the tool.

  5. Run the xsLogAnalyzer tool.

    The script is in the following locations:

    • In a stand-alone installation:

        wxs_install_root/ObjectGrid/bin

    • In an installation that is integrated with WebSphere Application Server:

        was_root/bin

    If your log files are large, consider using the -startTime, -endTime, and -maxRecords parameters when we run the report to restrict the number of log entries that are scanned. Using these parameters when we run the report makes the reports easier to read and run more effectively. We can run multiple reports on the same set of log files. 

    xsLogAnalyzer.sh -logsRoot c:\myxslogs -outDir c:\myxslogs\out 
                 -startTime 11.09.27_15.10.56.089 
                 -endTime 11.09.27_16.10.56.089 
                 -maxRecords 100

    -logsRoot Absolute path to the log directory to evaluate (required).
    -outDir Existing directory to write the report output. If you do not specify a value, the report is written to the root location of the xsLogAnalyzer tool.
    -startTime Start time to evaluate in the logs. The date is in the following format: year.month.day_hour.minute.second.millisecond
    -endTime End time to evaluate in the logs. The date is in the following format: year.month.day_hour.minute.second.millisecond
    -trace Trace string, such as

      ObjectGrid*=all=enabled
    -maxRecords Maximum number of records to generate in the report. The default is 100. If you specify 50, the first 50 records are generated for the specified time period.

  6. Open the generated files.

    If you did not define an output directory, the reports are generated in a folder called report_date_time. To open the main page of the reports, open the index.html file.

  7. Use the reports to analyze the log data. Use the following tips to maximize the performance of the report displays:

    • To maximize the performance of queries on the log data, use as specific information as possible. For example, a query for server takes much longer to run and returns more results than

      server_host_name.

    • Some views have a limited number of data points that are displayed at one time. We can adjust the segment of time that is being viewed by changing the current data, such as start and end time, in the view.


Create custom scanners for log analysis

We can create custom scanners for log analysis. After you configure the scanner, the results are generated in the reports when we run the xsLogAnalyzer tool. The custom scanner scans the logs for event records based on the regular expressions specified.

  1. Create a scanner specifications properties file that specifies the general expression to run for the custom scanner.

    1. Create and save a properties file.

      The file must be in directory...

        loganalyzer_root/config/custom

      We can name the file as: you like. The file is used by the new scanner, so naming the scanner in the properties file is useful, for example:

        my_new_server_scanner_spec.properties

    2. Include the following properties in the my_new_server_scanner_spec.properties file:

        include.regular_expression = REGULAR_EXPRESSION_TO_SCAN

      The REGULAR_EXPRESSION_TO_SCAN variable is a regular expression on which to filter the log files.

      Example: To scan for instances of lines that contain both the "xception" and "rror" strings regardless of the order, set the property...

        include.regular_expression

      ...to the following value:

        include.regular_expression = (xception.+rror)|(rror.+xception)

      This regular expression causes events to be recorded if the string "rror" comes before or after the "xception" string.

      Example: To scan through each line in the logs for instances of lines that contain either the phrase "xception" or the phrase "rror" strings regardless of the order, set the include.regular_expression property to the following value:

        include.regular_expression = (xception)|(rror)

      This regular expression causes events to be recorded if the either the "rror" string or the "xception" string exist.

  2. Create a configuration file that the xsLogAnalyer tool uses to create the scanner.

    1. Create and save a configuration file.

        The file must be in...

          loganalyzer_root/config/custom

        We can name the file as scanner_nameScanner.config, where scanner_name is a unique name for the new scanner. For example, you might name the file serverScanner.config

      • Include the following properties in the scanner_nameScanner.config file:

          scannerSpecificationFiles = loganalyzer_root/config/custom/my_new_scanner_spec.properties

        We can also specify multiple scanner specification files by using a semi-colon separated list:

          scannerSpecificationFiles = loganalyzer_root/config/custom/my_new_scanner_spec1.properties;loganalyzer_root/config/custom/my_new_scanner_spec2.properties

  3. Run the xsLogAnalyzer tool.


Results

After we run the xsLogAnalyzer tool, the report contains new tabs in the report for the custom scanners that you configured. Each tab contains the following views:

Charts

A plotted graph that illustrates recorded events. The events are displayed in the order in which the events were found.

Tables

A tabular representation of the recorded events.

Summary reports


Troubleshooting log analysis

Use the following troubleshooting information to diagnose and fix problems with the xsLogAnalyzer tool and its generated reports.


Troubleshooting the product installation

Installation Manager is a common installer for many IBM software products that you use to install this version of WXS.


Results

Notes on logging and tracing:

Notes on troubleshooting:

Note on version and history information: versionInfo.sh and historyInfo.sh return version and history information based on all of the installation, uninstallation, update, and rollback activities performed on the system.


JAVA: Troubleshooting client connectivity

There are several common problems specific to clients and client connectivity that we can solve as described in the following sections.


Troubleshooting cache integration

Use this information to troubleshoot issues with the cache integration configuration, including HTTP session and dynamic cache configurations.


JAVA: Troubleshooting the JPA cache plug-in

Use this information to troubleshoot issues with your JPA cache plug-in configuration. These problems can occur in both Hibernate and OpenJPA configurations.


10. Troubleshooting IBM eXtremeMemory

Use the following information to troubleshoot eXtremeMemory.

Problem: If the shared resource, libstdc++.so.5, is not installed, then when you start the container server, IBM eXtremeMemory native libraries do not load.

Linux: Symptom: On a Linux 64-bit operating system, if you try to start a container server with the enableXM server property set to true, and the libstdc++.so.5 shared resource is not installed, you get an error similar to the following example: 

00000000 Initialization W CWOBJ0006W: An exception occurred: java.lang.reflect.InvocationTargetException     
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)     
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)     
 at java.lang.reflect.Constructor.newInstance(Constructor.java:527)     
 at com.ibm.websphere.objectgrid.server.ServerFactory.initialize(ServerFactory.java:350)     
 at com.ibm.websphere.objectgrid.server.ServerFactory$2.run(ServerFactory.java:303)     
 at java.security.AccessController.doPrivileged(AccessController.java:202)     
 at com.ibm.websphere.objectgrid.server.ServerFactory.getInstance(ServerFactory.java:301)     
 at com.ibm.ws.objectgrid.InitializationService.main(InitializationService.java:302) 

Caused by: com.ibm.websphere.objectgrid.ObjectGridRuntimeException: java.lang.UnsatisfiedLinkError: 
 OffheapMapdbg (Not found in java.library.path)     
 at com.ibm.ws.objectgrid.ServerImpl.<init&gt;(ServerImpl.java:1033)     
... 9 more Caused by: java.lang.UnsatisfiedLinkError: OffheapMapdbg (Not found in java.library.path)     
 at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1011)     
 at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:975)    
 at java.lang.System.loadLibrary(System.java:469)     
 at com.ibm.ws.objectgrid.io.offheap.ObjectGridHashTableOH.initializeNative(ObjectGridHashTableOH.java:112)     
 at com.ibm.ws.objectgrid.io.offheap.ObjectGridHashTableOH.<clinit&gt;(ObjectGridHashTableOH.java:87)     
 at java.lang.J9VMInternals.initializeImpl(Native Method)     
 at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)     
 at com.ibm.ws.objectgrid.ServerImpl.<init&gt;(ServerImpl.java:1028)     
... 9 more

Cause: The shared resource

libstdc++.so.5 has not been installed.

Diagnosing the problem: To verify that the resource

libstdc++.so.5 is installed, issue the following command from the ObjectGrid/native directory of the installation: 

ldd libOffheapMap.so
If you do not have the shared library installed, you get the following error: 
ldd libOffheapMap.so libstdc++.so.5 => not found

Resolving the problem: Use the package installer of your 64-bit Linux distribution to install the required resource file. The package might be listed as

compat-libstdc++-33.x86_64 or libstdc++5. After installing the required resource, verify that the libstdc++5package is installed by issuing the following command from the ObjectGrid directory of the installation: 

ldd libOffheapMap.so


11. Troubleshooting administration

Use the following information to troubleshoot administration, including starting and stopping servers, using the xscmd utility, and so on.


12. Troubleshooting data monitoring

Use this information to troubleshoot monitoring activities that you complete with the WebSphere eXtreme Scale web console or other utilities to monitor the performance of the application environment.

Problem: We cannot switch between domains with different security settings in the WebSphere eXtreme Scale web console.

We can switch domains between two unsecure domains. We can also switch domains between two secure domains with the same security configured. However, we cannot switch between one unsecure and one secure domain or between two secure domains with different security settings.

Diagnosis: The startOgServer command is used to start two different catalog servers in separate domains. Each catalog server is unaware of the other. However, both catalog servers are started with the same domain name. When you do not specify the domain name, both catalog servers start in different domains with the default name, false Domain. In addition, the monitoring console displays data for only one of the catalog server domains.

Cause: When you switch domains in the monitoring console, we are connected to the second domain. However, no grid data from that domain is displayed, and the first domain grid data is still in view. Therefore, during run time, both catalog servers run in separate domains with the name, false Domain.

Solution: Determine which domain names are used when catalog servers start in the two domains. To identify the domain names, analyze your startOgServer command syntax and investigate what domain is being specified.

Since this problem scenario is not supported, complete the following actions to display the correct catalog service domain statistics:

  1. Shut down the catalog servers, and verify that they are configured to start with unique domain names.

  2. Restart your monitor console.

  3. Optional: If an outage is not possible, consider running a second monitoring console to monitor the second domain.


13. Troubleshooting multiple data center configurations

Use this information to troubleshoot multiple data center configurations, including linking between catalog service domains.

You must use the xscmd utility to troubleshoot your multiple data center configurations.

Improve response time and data availability with WebSphere eXtreme Scale multi-master capability

com.ibm.websphere.objectgrid.openJPA package

com.ibm.websphere.objectgrid.hibernate.cache package


14. JAVA: Troubleshooting loaders

Use this information to troubleshoot issues with your database loaders.


15. Troubleshooting XML configuration

When you configure eXtreme Scale, we can encounter unexpected behavior with your XML files. The following sections describe problems that can occur and solutions.

Related reference:

Configuration files

ObjectGrid descriptor XML file

Deployment policy descriptor XML file

Entity metadata descriptor XML file

Security descriptor XML file

Client properties file

Spring descriptor XML file


16. Troubleshooting deadlocks

The following sections describe some of the most common deadlock scenarios and suggestions on how to avoid them.

Implement exception handling in the application.

The following exception displays as a result: 

com.ibm.websphere.objectgrid.plugins.LockDeadlockException: Message
This message represents the string that is passed as a parameter when the exception is created and thrown.


17. JAVA: Troubleshooting lock timeout exceptions for a multi-partition transaction

The scenario that is described is an example of a multi-partition transaction that is causing a lock timeout exception. Depending on the state of the transaction, the solutions illustrate how we can manually resolve this problem.

Implement exception handling in the application.

The following exception displays as a result: 

Caused by: com.ibm.websphere.objectgrid.LockTimeoutException: 
Local-40000139-DEF8-05EA-E000-64A856931719 timed out waiting 
for lock mode S to be granted for map name: TS2_MapP, key: key12
granted = X
lock request queue ->[WXS-40000139-DEF6-FA84-E000-1CB456931719, state = Granted, requested 
73423 milli-seconds ago, marked to keep current mode false, 
snapshot mode 0, mode = X, thread name = xIOReplicationWorkerThreadPool : 29]
->[Local-40000139-DEF8-05EA-E000-64A856931719, state 
= Waiting for 5000 milli-seconds, marked to keep current mode false, 
snapshot mode 0, mode = S, thread name = xIOWorkerThreadPool : 28]
dump of all locks for WXS-40000139-DEF6-FA84-E000-1CB456931719
Key: key12, map: TS2_MapP
strongest currently granted mode for key is X
->[WXS-40000139-DEF6-FA84-E000-1CB456931719, state = Granted, 
requested 73423 milli-seconds ago, marked to keep current mode false, 
snapshot mode 0, mode = X, thread name = xIOReplicationWorkerThreadPool : 29]
dump of all locks for Local-40000139-DEF8-05EA-E000-64A856931719
This message represents the string that is passed as a parameter when the exception is created and thrown.

Problem: You see a lock timeout exception and the holder of the lock is a multi-partition transaction, or, the log folder is increasing with log messages.

Diagnosis:

You will see a log messages repeatedly filling up your log folder such as the following: 

00000099 TransactionLog I  CWOBJ8705I:
Automatic resolution of transaction 
WXS-40000139-DF01-216D-E002-1CB456931719 
at RM:TestGrid:TestSet2:20 is still waiting for a decision. 
Another attempt to resolve the transaction will occur in 30 seconds.
Determine what type of transaction is causing the lock. If the prefix on the transaction identifier is WXS-, then is indicates multi-partition transaction. If the prefix on the transaction identifier is Local-, then this indicates that the transaction is single partition transaction.

Cause: The application is likely holding the lock because a commit or rollback did not occur.

Solution: Determine the state of the transaction and how long it was in that state. Use either the command utility xscmd -c listindoubts with option -d (for a detailed output) or use the transaction MBean.


JAVA: Resolving lock timeout exceptions

Using the xscmd -c listindoubt command, we can view the state of a transaction and determine a course of action.

JAVA: Troubleshooting lock timeout exceptions for a multi-partition transaction


Resolving lock timeout exceptions with the xscmd -c listindoubts command

Procedure


18. Troubleshooting security

Use this information to troubleshoot issues with your security configuration.


19. Troubleshooting Liberty profile configurations

Use this information to troubleshoot commonly experienced problems with the Liberty profile.

To help you identify and resolve problems, the product has a unified logging component.

Details of known restrictions that apply when using the Liberty profile are provided in the following two topics in the WebSphere Application Server Information Center:


20. Collecting data with the IBM Support Assistant Data Collector

Run the IBM Support Assistant Data Collector to collect problem determination data from the WXS environment. By using this tool, we can reduce the amount of time it takes to reproduce a problem with the proper RAS tracing levels set, and reduce the effort required to send the appropriate log information to IBM Support.

Before we run the tool, have the following system configuration information ready to provide to the tool:

In previous releases of WXS, the IBM Support Assistant Lite tool was used for log gathering for problem determination. The IBM Support Assistant Lite tool continues to be shipped with the product in the wxs_home /isalite_wxs directory. IBM Support Assistant Data Collector is a more interactive tool that installs with Version 8.6 and later. IBM Support Assistant Data Collector improves ease of use of collecting data by remembering various inputs, reducing repetitive typing during console input. For more information, see IBM Support Assistant Data Collector .

  1. Start the tool. The tool runs in console mode by starting the launch script from the command line. The script for the tool is installed in the wxs_home /isalite_dc directory.

    • WINDOWS: isadc.bat
    • UNIX/Linux: isadc.sh

  2. Supply your system information to the tool. At each step, the choices are presented as numbered lists and you input the number of your selection and press the enter key. When input is required, prompts are displayed at which you enter your response and press the enter key. We can find collection details for each problem type in their corresponding MustGather documents. You also can provide the compressed file name and the directory location to which we want to save your bundled information.

  3. Stop the collector tool by typing the quit option in console mode.


Results

The following environment-related information is bundled in a compressed file that you named for saving the data:


What to do next

Contact IBM support and provide the compressed file that you generated with the IBM Support Assistant Data Collector.


21. IBM Support Assistant for WebSphere eXtreme Scale

Use the IBM Support Assistant to collect data, analyze symptoms, and access product information.


IBM Support Assistant Lite

IBM Support Assistant Lite for WebSphere eXtreme Scale provides automatic data collection and symptom analysis support for problem determination scenarios.

IBM Support Assistant Lite reduces the amount of time it takes to reproduce a problem with the proper Reliability, Availability, and Serviceability tracing levels set (trace levels are set automatically by the tool) to streamline problem determination. If you need further assistance, IBM Support Assistant Lite also reduces the effort required to send the appropriate log information to IBM Support.

IBM Support Assistant Lite is included in each installation of WXS Version 7.1.0


IBM Support Assistant

IBM Support Assistant (ISA) provides quick access to product, education, and support resources that can help you answer questions and resolve problems with IBM software products on your own, without needing to contact IBM Support. Different product-specific plug-ins let you customize IBM Support Assistant for the particular products you have installed. IBM Support Assistant can also collect system data, log files, and other information to help IBM Support determine the cause of a particular problem.

IBM Support Assistant is a utility to be installed on your workstation, not directly onto the WebSphere eXtreme Scale server system itself. The memory and resource requirements for the Assistant could negatively affect the performance of the WebSphere eXtreme Scale server system. The included portable diagnostic components are designed for minimal impact to the normal operation of a server.

Use IBM Support Assistant to help you in the following ways:

Finally, we can use the built-in Updater facility to obtain support for additional software products and capabilities as they become available. To set up IBM Support Assistant for use with WebSphere eXtreme Scale, first install IBM Support Assistant using the files provided in the downloaded image from the IBM Support Overview Web page at: http://www-947.ibm.com/support/entry/portal/Overview/Software/Other_Software/IBM_Support_Assistant . Next, use IBM Support Assistant to locate and install any product updates. We can also choose to install plug-ins available for other IBM software in your environment. More information and the latest version of the IBM Support Assistant are available from the IBM Support Assistant Web page at: http://www.ibm.com/software/support/isa/ .