Home

Monitor WebSphere eXtreme Scale v8.6

+

Search Tips   |   Advanced Search

  1. Statistics overview
  2. Monitor with the web console
    1. Start and log on to the web console
    2. Connect the web console to catalog servers
    3. View statistics with the web console
    4. Web console statistics
    5. Monitor with custom reports
  3. Monitor the health of the environment
    1. Message center overview
    2. Configure the message center
    3. View health event notifications in the message center
    4. View health notifications with the xscmd utility
  4. Monitor with CSV files
    1. CSV file statistics definitions
  5. Enable statistics
    1. Statistics modules
    2. Monitor with the statistics API
    3. Monitor with the xscmd utility
  6. Monitor with WAS PMI
    1. Enable PMI
    2. Retrieve PMI statistics
    3. PMI modules
    4. Access MBeans using wsadmin
  7. Monitor server statistics with MBeans
  8. Monitor client HTTP session statistics
  9. Monitor with vendor tools
    1. Monitor with the IBM Tivoli Enterprise Monitoring Agent for WXS
    2. Monitor applications with CA Wily Introscope
    3. Monitor with Hyperic HQ
  10. Monitor eXtreme Scale information in DB2


Statistics overview

Statistics in WebSphere eXtreme Scale are built on an internal statistics tree. The StatsAccessor API, PMI modules, and MBean API are built from the internal tree.

Each of these APIs offer a view into the statistics tree, but are used for different reasons:

Statistics API Direct access to statistics for custom MBeans or logging.
MBean API Uses the Statistics API. Runs local to the server JVM. Use for distributed object grids.
WAS PMI modules Use if we are running WXS within WAS. Provides view of the internal statistics tree.


Statistics API

Much like a tree map, there is a corresponding path and key used to retrieve a specific module, or in this case granularity or aggregation level. For example, assume there is always an arbitrary root node in the tree and that statistics are being gathered for a map named "payroll," belonging to an ObjectGrid named "accounting." For example, to access the module for a map's aggregation level or granularity, you could pass in a String[] of the paths. In this case that would equate to String[] {root, "accounting", "payroll"}, as each String would represent the node's path. The advantage of this structure is that a user can specify the array to any node in the path and get the aggregation level for that node. So passing in String[] {root, "accounting"} would give you map statistics, but for the entire grid of "accounting." This leaves the user with both the ability to specify types of statistics to monitor, and at whatever level of aggregation is required for the application.


WebSphere Application Server PMI modules

WebSphere eXtreme Scale includes statistics modules for use with the WebSphere Application Server PMI. When a WAS profile is augmented with WXS, the augment scripts automatically integrate the WXS modules into the WAS configuration files.

With PMI, we can...


Vendor product integration with Managed Beans (MBean)

JConsole or MC4J are some examples of lightweight JMX consoles that can be used to analyze information about a WXS topology. We can also use the programmatic APIs to write adapter implementations to snapshot or track eXtreme Scale performance. WXS includes a sample monitoring application that allows out-of-the box monitoring capabilities, and can be used as a template for writing more advanced custom monitoring utilities.


Monitoring with the web console

With the web console, we can chart current and historical statistics. This console provides some preconfigured charts for high-level overviews, and has a custom reports page that we can use to build charts from the available statistics. Use the charting capabilities in the monitoring console of WXS to view the overall performance of the data grids in your environment.


Start web console

To start the console server.

To log on to the web console...

Default user/pass is admin/admin

Recommended browsers

To run the console server on a port other than the default, edit...

Default port for the console server is 7080 for HTTP and 7443 for HTTPS.

Restart the server to use the new port numbers.

To edit the console configuration...

The console configuration includes information such as:

Connect the catalog servers to the web console to start tracking statistics.

To stop the web console server...


Connect the web console to catalog servers

  1. Start the web console server

  2. Start at least one catalog server

  3. If the catalog servers have SSL enabled, configure a keystore, truststore, and client properties file.

    SSL is enabled for a catalog server by setting the transportType attribute to SSL-Required in the Server properties file .

    1. Configure a keystore and truststore, and then exchange, or cross-import the public certificates.

      For example, copy the truststore and keystore to a location on the server running the web console.

    2. Edit the client properties file on the web console server to include the properties for SSL configuration. For example, edit...

        /path/to/ObjectGridProperties/sampleclient.properties

      The following properties are required for outbound SSL connections from the web console: 

       
#------------------------------------------------------------------------------
 # SSL Configuration #
 # - contextProvider     (IBMJSSE2, IBMJSSE, IBMJSSEFIPS, etc.)
 # - protocol            (SSL,  SSLv3, TLS, TLSv1, etc.)
 # - keyStoreType        (JKS, JCEK, PKCS12, etc.)
 # - trustStoreType      (JKS, JCEK, PKCS12, etc.)
 # - keyStore            (fully qualified path to key store file)
 # - trustStore          (fully qualified path to trust store file)
 # - alias               (string specifying ssl certificate alias to use from keyStore)
 # - keyStorePassword    (string specifying password to the key store - encoded or not)
 # - trustStorePassword  (string specifying password to the trust store - encoded or not)
 #
 # Uncomment these properties to set the SSL configuration.
 #------------------------------------------------------------------------------
 #alias=clientprivate #contextProvider=IBMJSSE
 #protocol=SSL
 #keyStoreType=JKS
 #keyStore=etc/test/security/client.private #keyStorePassword={xor}PDM2OjErLyg\=
 #trustStoreType=JKS
 #trustStore=etc/test/security/server.public #trustStorePassword={xor}Lyo9MzY8

      If we are using Windows, you must escape any backslash (

      \ ) characters in the path. For example, to use the path C:\opt\ibm, enter C:\\opt\\ibm in the properties file.

  4. Establish and maintain connections to catalog servers to monitor. Repeat the following steps to add each catalog server to the configuration.

    1. Click Settings > eXtreme Scale Catalog Servers.

    2. Add a new catalog server.

      1. Click the add icon () to register an existing catalog server.
      2. Provide information, such as the host name and listener port.
      3. Click OK.
      4. Verify that the catalog server has been added to the navigation tree.

  5. Group the catalog servers that you created into a catalog service domain. Create a catalog service domain when security is enabled in the catalog servers because security settings are configured in the catalog service domain.

    1. Click Settings > eXtreme Scale Domains page.

    2. Add a new catalog service domain.

      1. Click the add icon () to register a catalog service domain. Enter a name for the catalog service domain.

      2. After you create the catalog service domain, we can edit the properties. The catalog service domain properties follow:

        Name

        Indicates the host name of the domain, as assigned by the administrator.

        Catalog servers

        List one or more catalog servers that belong to the selected domain. We can add the catalog servers that you created in the previous step.

        Generator class

        Name of the class that implements the CredentialGenerator interface. This class is used to get credentials for clients. If you specify a value in this field, the value overrides the crendentialGeneratorClass property in the client.properties file.

        Generator properties

        Properties for the CredentialGenerator implementation class. The properties are set to the object with the setProperties(String) method. The credentialGeneratorprops value is used only if the value of the credentialGeneratorClass property is not null. If you specify a value in this field, the value overrides the credentialGeneratorProps property in the client.properties file.

        eXtreme Scale client properties path

        Path to the client properties file that you edited to include security properties in a previous step. For example, you might indicate the c:\ObjectGridProperties\sampleclient.properties file. To stop the console from trying to use secure connections, we can delete the value in this field. After you set the path, the console uses an unsecured connection.

      3. Click OK.

      4. Verify that the domain has been added to the navigation tree.

    To view information about an existing catalog service domain, click the name of the catalog service domain in the navigation tree on the Settings > eXtreme Scale Domains page.

  6. View the connection status. The Current domain field indicates the name of the catalog service domain that is currently being used to display information in the web console. The connection status displays next to the name of the catalog service domain.


Viewing statistics with the web console

We can monitor statistics and other performance information with the web console.

Before we can view statistics with the web console, you must complete the following tasks:

  1. Start the web console server.

  2. Connect the catalog servers to the web console server.

  3. Run active data grids and applications within the servers that are managed by the catalog service domain.

After you create your data grids and configure the applications to use the data grids, allow some time to pass for the statistics to become available. For example, with a dynamic cache data grid, statistics are not available until a WebSphere Application Server running a dynamic cache connects to the dynamic cache. In general, wait up to one minute after a major configuration change to see the changes in your statistics.

To view more specific information about any data point in a chart, we can move the mouse pointer over the data point.


Web console statistics

Depending on the view we are using in the web console, we can view different statistics about the configuration. These statistics include the used memory, the top used data grids, and the number of cache entries.


Data grid domain overview

Data grid domain overview statistics are displayed on the Monitor > Data Grid Domain Overview page. Click one of the following tabs for more information about the data grid domain:

Used Capacity tab

In the Current Data Grid Used Capacity Distribution chart, a picture of the Total Pool, and the Largest Used Capacity Consumers are displayed. Only the top 25 data grids are displayed. In the Used Capacity Over Time chart, the number of bytes that are consumed by the data grid is displayed.

Average Throughput tab

The 5 Most Active Data Grids by Average Transaction Time in Milliseconds chart contains a list of the top five data caches, organized by the average transaction time. The Average Throughput Over time chart displays the average, maximum, and minimum throughput within the last hour, day, and week.

Average Transaction Time tab

The 5 Slowest Data Grids chart displays data about the slowest data grids. The Average Transaction Time Over Time chart displays the average, maximum, and minimum transaction time within the last hour, day, and week.


Data grid overview

To view statistics for an individual data grid...

Current summary over last 30 seconds

Display the current number of cache entries, average transaction time, average throughput, and cache hit rate for the selected data grid.

Used Capacity tab

The Current summary over last 30 seconds chart displays the number of cache entries and used capacity in bytes over a specified time range.

Cache Usage tab

The Cache Usage chart helps to visualize the number of successful queries to the cache, and displays cache attempts, cache hits, and the cache hit rate over a specified time range.

Average Throughput tab

The Average Throughput vs. Average Transaction Time chart displays the transaction time and throughput over a specified time range.


Data grid details

Data grid statistics are displayed on the Monitor > Data Grid Details page. We can look at data for a selected grid and the maps that are within that grid.

Current summary over last 30 seconds

Display the current used capacity, number of cache entries, average throughput, and average transaction time for the selected data grid.

Current eXtreme Scale Object Grid Map Used Capacity Distribution

View a total pool, which includes the capacity by zone and the total capacity in each zone. Only the top 25 ObjectGrid maps are displayed. We can also view the largest used capacity consumers by each map.

Current Zone Used Capacity Distribution

View a total pool, which includes the total pool and the top used capacity consumers in the zone of the selected data grid. We can also view the largest used capacity consumers by each zone.

Map statistics:

Current summary over last 30 seconds

Display the current used capacity, number of cache entries, average throughput, and average transaction time for the selected map.

Current Partition Used Capacity Distribution

View a partition, which includes the total pool and the top used capacity consumers. Only the top 25 partitions are displayed. We can also view the largest used capacity consumers by each partition.


Server overview

Server statistics are displayed on the Monitor > Server Overview page.

Current Server Used Memory Distribution

This chart is composed of two views. Total Pool displays the current amount of used (real) memory in the server run time. Largest Used memory Consumers breaks down the used memory by server; however only the top 25 servers that are using the most memory are displayed.

Total Memory Over Time
Displays the real memory usage in the server run time.

Used Memory Over Time

Display the amount of used memory in the server run time.


Custom reports: Catalog service domain statistics

We can view catalog service domain statistics by creating a custom report. Click Monitor > Custom Reports.

Average Transaction Time (ms)

Display the average time required to complete a transaction in this domain.

Average Transaction Throughput (trans/sec)

Display the average number of transactions per second in this domain.

Maximum Transaction Time (ms)

Display the time spent by the most time-consuming transaction in this domain.

Minimum Transaction Time (ms)

Display the time spent by the least time-consuming transaction in this domain.

Total Transaction Time (ms)

Display total time spent on transactions in this domain, since the time the domain was initialized.


Custom reports: Container server statistics

We can view container server statistics by creating a custom report. Click Monitor > Custom Reports.

Average Transaction Time (ms)
Displays the average time required to complete a transaction for this catalog server.

Average Transaction Throughput (trans/sec)
Displays the average number of transactions per second for this catalog server.

Maximum Transaction Time (ms)
Displays the time spent by the most time-consuming transaction for this catalog server.

Minimum Transaction Time (ms)
Displays the time spent by the least time-consuming transaction for this catalog server.

Total Transaction Time (ms)
Displays total time spent on transactions for this catalog server, since the time for this catalog server was initialized.

Total Entries in Cache

Display the current number of objects cached in the grids overseen by this catalog server.

Hit rate (percentage)

Display the hit rate (hit ratio) for the selected data grid. A high hit rate is desirable. The hit rate indicates how well the grid is helping to avoid accessing the persistent store.

Used Bytes
Displays memory consumption by this map. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

Minimum Used Bytes
Displays the low point in memory consumption by this catalog service and its maps. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

Maximum Used Bytes
Displays the high point in memory consumption by this catalog service and its maps. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

Total Number of Hits
Displays the total number of times the requested data was found in the map, avoiding the need to access persistent store.

Total Number of Gets
Displays the total number of times the map had to access the persistent store to obtain data.

Free Heap (MB)

Display the actual amount of heap available to the JVM being used by the catalog server.

Total Heap

Display the amount of heap available to the JVM being used by this catalog server.

Number of Available Processors
Displays the number of processors that are available to this catalog service and its maps. For the highest stability, run your servers at 60% processor loading and JVM heaps at 60% heap loading. Spikes can then drive the processor usage to 80.90%, but do not regularly run your servers higher than these levels

Maximum Heap Size (MB)

Display the maximum amount of heap available to the JVM being used by this catalog server.

Used Memory

Display the used memory in the JVM being used by this catalog server.


Custom reports: Data grid statistics

We can view data grid statistics by creating a custom report. Click Monitor > Custom Reports.

Average Transaction Time (ms)

Display the average time required to complete transactions involving this grid.

Average Transaction Throughput (trans/sec)

Display the average number of transactions per second completed by this grid.

Maximum Transaction Time (ms)

Display the time spent by the most time-consuming transaction completed by this grid.

Minimum Transaction time (ms)

Display the time spent by the least time-consuming transaction completed by this grid.

Total Transaction Time (ms)

Display the total amount of transaction processing time for this grid.


Custom reports: Map statistics

We can view map statistics by creating a custom report. Click Monitor > Custom Reports.

Total Entries in Cache

Display the current number of objects cached in this map.

Hit Rate (percentage)

Display the hit rate (hit ratio) for the selected map. A high hit rate is desirable. The hit rate indicates how well the map is helping to avoid accessing the persistent store.

Used Bytes

Display memory consumption by this map. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

Minimum Used Bytes

Display the minimum consumption (in Bytes) for this map. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

Maximum Used Bytes

Display the maximum consumption (in Bytes) for this map. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

Total Number of Hits

Display the total number of times the requested data was found in the map, avoiding the need to access persistent store.

Total Number of Gets

Display the total number of times the map had to access the persistent store to obtain data.

Free Heap (MB)

Display the current amount of heap available to this map, in the JVM being used by the catalog server.

Total Heap (MB)

Display the total amount of heap available to this map, in the JVM being used by the catalog server. For the highest stability, run your servers at 60% processor loading and JVM heaps at 60% heap loading. Spikes can then drive the processor usage to 80.90%, but do not regularly run your servers higher than these levels

Number of Available Processors

Display the number of processors available to this map. For the highest stability, run your servers at 60% processor loading and JVM heaps at 60% heap loading. Spikes can then drive the processor usage to 80.90%, but do not regularly run your servers higher than these levels

Maximum Heap Size (MB)

Display the maximum amount of heap available to this map, in the JVM being used by the catalog server.

Used Memory (MB)

Display the used amount of memory in this map.


Monitoring with custom reports

We can build custom reports to save various charts that contain statistics about the catalog service domains, data grids, and container servers in your environment. We can save the custom reports and load them to view again later.

Before we can view statistics with the web console, you must complete the following tasks:

  1. Start the web console server.

  2. Connect the catalog servers to the web console server.

  3. Run active data grids and applications within the servers that are managed by the catalog service domain.


Monitoring the health of the environment

The message center provides an aggregated view of event notifications for log and first-failure data capture (FFDC) messages. We can view these event notifications with the message center in the web console, the xscmd utility, or programmatically with MBeans.


Message center overview

The message center aggregates health status events from all container and catalog servers in a catalog service domain, in real time. When the message center is configured, we can view an overview of the current critical events that are occurring in various servers without collecting the logs for each server.


Message center implementation

Data grid deployments can involve dozens or hundreds of distributed server processes. If a problem occurs, we can open the actual log file for the affected container server to further analyze the problem. The message center consists of the following components:

Event aggregation

When you configure health monitoring on a catalog server, you receive aggregated events that are affecting the health of the entire catalog service domain. The framework includes an indication of the source and severity for the following types of events:

  • All FFDC events
  • All WARNING or SEVERE log entries
  • A filtered list of all log entries, including INFO, WARNING, and SEVERE log entries
  • Server start and server stop operations
  • Loss or regaining of quorum

Message center in the web console

The message center in the web console displays the aggregated event records. These events include both recent events and real-time update notifications for events that occurred after the console was opened.

Events in the xscmd utility

We can also display a recent list of events with the xscmd utility. As events occur, we can redirect the event records to create automatic scripting utilities.

MBeans for integration with other monitoring software

We can also use the available management MBeans to plug the message center into your other JMX monitoring software. The documentation for these MBeans is included in the API documentation.


Message center versus log analyzer

The log analyzer is another tool to analyze a set of log messages. This tool requires that you manually collect the logs from the various servers in your environment. Then, we can run the tool to create reports of problem conditions. Use the log analyzer for post-mortem analysis of your logs when you need to analyze a set of messages that is larger than the subset of 1000 messages that we can display in the message center. Use the message center for real-time monitoring of the health of the data grid to quickly identify issues that are occurring. Then, we can either review the log files for the related container server, or use the log analyzer to further research the problem.


Health monitoring configuration and architecture

We can enable the message center by configuring one or more catalog servers as a hub. Each hub has its own subscriptions and separate event histories. Each event in the history has a sequence number. The event histories on separate catalog servers are not kept synchronized and are different. Catalog servers can subscribe to log and FFDC events from other catalog servers.


Configure the message center

To use the message center, configure the catalog servers as messaging hubs.

  1. Activate the catalog server as a hub for the health monitoring framework. All catalog servers are activated as hubs by default. We can enable or disable this setting with the following property in the server.properties file for the catalog server:

    enableManagementConcentrator

    Specifies if the catalog server is a hub for the message center. This property is enabled by default. To disable the hub, set the value to false.

    Default: true

  2. Optional: If we want INFO log messages to be included, specify a regular expression that filters the INFO log messages. Specify your regular expression with the following property in the server.properties file for the catalog server:

    logNotificationFilter

    Specifies a regular expression that filters all messages, including the INFO level log messages. This filter determines which messages generate health monitoring events. If you do not specify a regular expression, INFO level log messages are not published through the health monitoring framework. By default, only WARNING and SEVERE level messages generate health monitoring events.

    Example: logNotificationFilter=.*DYNACACHE.*

  3. When you change server properties, restart the catalog server.


What to do next

After the catalog servers are activated as hubs for the health monitoring framework, we can use the message center in the web console or the xscmd utility to view notifications for health events.


Viewing health event notifications in the message center

Use the message center in the web console to assess the real-time health of the entire data grid and catalog service domain. The events that are displayed in the message center are a subset of events that are filtered to display the most critical issues.


What to do next

If you notice that critical events are occurring on one of your container servers, open the log file for the container server for further analysis.

java.utils.regex.Pattern API documentation


Viewing health notifications with the xscmd utility

We can view current event notifications, show event notification history, and set notification filters from the message center with the xscmd utility.


Monitoring with CSV files

We can enable monitoring data collected for a container server to be written to comma-separated values (CSV) files. These CSV files can contain information about the JVM, map, or ObjectGrid instance.

By enabling monitoring data to be written to CSV files, we can download and analyze historical data for individual container servers. Data begins being collected when you start the server with the server properties that enable the CSV files. We can then download the CSV files at any time and use the files as you choose.

  1. Update the server properties file with the following properties that are related to enabling the CSV files. 
    parameter=default value jvmStatsLoggingEnabled=true maxJVMStatsFiles=5
maxJVMStatsFileSize=100
jvmStatsFileName=jvmstats jvmStatsWriteRate=10

mapStatsLoggingEnabled=true maxMapStatsFiles=5
maxMapStatsFileSize=100
mapStatsFileName=mapstats mapStatsWriteRate=10

ogStatsLoggingEnabled=true maxOGStatsFiles=5
maxOGStatsFileSize=100
ogStatsFileName=ogstats ogStatsWriteRate=10

  2. Restart the server to pick up the changes to the server properties file.

  3. Download the CSV file. The CSV file is written to the server_name/logs directory.

    Each CSV file contains a header that labels each of the columns. Each column is delineated by a comma.

  4. Import the CSV file into the program that we are using to process the data, such as a spreadsheet.


CSV file statistics definitions

The CSV files that we can download for a server include statistics that we can use to build historical charts or other information.


JVM statistics log

TimeStamp (column 1)

Specifies the date and time of the statistics snapshot that was taken for the JVM.

ServerName (column 2)

Server name of the JVM.

Hostname (column 3)

Host name of the JVM.

FreeMemory (column 4)

Number of available bytes for the JVM.

MaxMemory (column 5)

Maximum number of bytes that can be allocated for the JVM.

TotalMemory (column 6)

Display the real memory usage in the server run time.

AvailProcs (column 7)

Display the number of processors that are available to this catalog service and its maps. For the highest stability, run your servers at 60% processor loading and JVM heaps at 60% heap loading. Spikes can then drive the processor usage to 80.90%, but do not regularly run your servers higher than these levels


Map statistics log

TimeStamp (column 1)

Specifies the date and time of the statistics snapshot that was taken for the map.

MapName (column 2)

Name of the map.

OgName (column 3)

Name of the data grid to which this map belongs.

PartitionId (column 4)

ID of the partition.

MapSetName (column 5)

Map set to which this map belongs.

HitRate (column 6)

Display the hit rate (hit ratio) for the selected map. A high hit rate is desirable. The hit rate indicates how well the data grid is helping to avoid accessing the persistent store.

Count (column 7)

Indicates a count of the data samples that have been gathered since the server started. For example, a value of 100 indicates that the entry is the 100th sample entry that has been gathered since the server started.

TotalGetCount (column 8)

Display the total number of times the map had to access the persistent store to obtain data.

TotalHitCount (column 9)

Display the total number of times the requested data was found in the map, avoiding the need to access persistent store.

StartTime (column 10)

Specifies the time that the counters began from last reset call. The resets occur when the server starts or restarts.

LastCount (column 11)

Amount of time since the last data sample was taken.

LastTotalGetCount (column 12)

Indicates the current total number of get operations from the cache minus the number of get operations in the previous time period.

LastTotalHitCount (column 13)

Indicates the current total number of hits from the cache minus the number of hits in the previous time period.

UsedBytes (column 14)

Display memory consumption by this map. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

MinUsedBytes (column 15)

Display the low point in memory consumption by this catalog service and its maps. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

MaxUsedBytes (column 16)

Display the high point in memory consumption by this catalog service and its maps. The used bytes statistics are accurate only when we are using simple objects or the COPY_TO_BYTES copy mode.

LastUsedBytes (column 17)

Indicates the current UsedBytes value minus the UsedBytes value from the previous statistics collection period.

SampleLen (column 18)

Indicates the length, in milliseconds, of the time period during with the data was sampled.


ObjectGrid statistics log

TimeStamp (column 1)

Specifies the date and time of the statistics snapshot that was taken for the data grid.

OgName (column 2)

Name of the data grid.

PartitionId (column 3)

Partition ID.

Count (column 4)

Indicates a count of the data samples that have been gathered since the server started. For example, a value of 100 indicates that the entry is the 100th sample entry that has been gathered since the server started.

Hostname (column 5)

Host name.

DomainName (column 6)

Catalog service domain to which this data grid belongs.

MaxTime (column 7)

Display the time spent by the most time-consuming transaction for this server.

MinTime (column 8)

Display the time spent by the least time-consuming transaction for this server.

MeanTime (column 9)

Average time spent on a transaction.

TotalTime (column 10)

Display total time spent on transactions for this server, since the time for this server was initialized.

AvgTransTime (column 11)

Display the average time required to complete a transaction for this server.

AvgThroughPut (column 12)

Display the average number of transactions per second for this server.

SumOfSquares (column 13)

Sum of squares value for the transaction time. This value measures the deviation from the mean at the given point in time.

SampleLen (column 14)

Indicates the length, in milliseconds, of the time period during with the data was sampled.

LastDataSample (column 15)

Specifies the time since the last sample was taken.

LastTotalTime (column 16)

Current total time minus the previous total time for the data sample.

StartTime (column 17)

Indicates the time that the statistics began to be collected since the last reset of the data. The data is reset when the server restarts.


Enable statistics

WebSphere eXtreme Scale uses an internal statistics model to track and filter data, which is the underlying structure that all data views use to gather snapshots of statistics. Use several methods to retrieve the information from the statistics modules.

For a list of all of the modules on which we can enable statistics, see StatsSpec class .


Example

Some examples of statsSpec strings that you might specify using the properties file, xscmd utility, or StatsSpec interface follow: Enable all statistics for all modules: 

all=enabled
Disable all statistics for all modules: 
all=disabled
Enable statistics for all statistics in the OGStatsModule: 
og.all=enabled
Enable statistics for all statistics in the OGStatsModule and MapStatsModule: 
og.all=enabled;map.all=enabled
Enable statistics for only Map Used bytes statistic, and disable everything else: 
all=disabled;map.usedbytes=enabled


Statistics modules

WebSphere eXtreme Scale uses an internal statistics model to track and filter data, which is the underlying structure that all data views use to gather snapshots of statistics.


Overview

Statistics in WebSphere eXtreme Scale are tracked and contained within StatsModules components. Within the statistics model, several types of statistics modules exist:

OGStatsModule

Provides statistics for an ObjectGrid instance, including transaction response times.

MapStatsModule

Provides statistics for a single map, including the number of entries and hit rate.

QueryStatsModule

Provides statistics for queries, including plan creation and run times.

AgentStatsModule

Provides statistics for DataGrid API agents, including serialization times and run times.

HashIndexStatsModule

Provides statistics for HashIndex query and maintenance run times.

SessionStatsModule

Provides statistics for the HTTP session manager plug-in.
For details about the statistics modules, see the Statistics API .


Statistics in a local environment

The model is organized like an n-ary tree (a tree structure with the same degree for all nodes) comprised of all of the StatsModule types mentioned in the previous list. Because of this organization structure, every node in the tree is represented by the StatsFact interface. The StatsFact interface can represent an individual module or a group of modules for aggregation purposes. For example, if several leaf nodes in the tree represent particular MapStatsModule objects, the parent StatsFact node to these nodes contains aggregated statistics for all of the children modules. After you fetch a StatsFact object, we can then use interface to retrieve the corresponding StatsModule.

Much like a tree map, you use a corresponding path or key to retrieve a specific StatsFact. The path is a String[] value that consists of every node that is along the path to the requested fact. For example, you created an ObjectGrid called ObjectGridA, which contains two Maps: MapA and MapB. The path to the StatsModule for MapA would look like

[ObjectGridA, MapA]. The path to the aggregated statistics for both maps would be:

[ObjectGridA].


Statistics in a distributed environment

In a distributed environment, the statistics modules are retrieved using a different path. Because a server can contain multiple partitions, the statistics tree needs to track the partition to which each module belongs. As a result, the path to look up a particular StatsFact object is different. Using the previous example, but adding in that the maps exist within partition 1, the path is

[1, ObjectGridA, MapA] for retrieving that StatsFact object for MapA.


Monitoring with the statistics API

The Statistics API is the direct interface to the internal statistics tree. Statistics are disabled by default, but can be enabled by setting a StatsSpec interface. A StatsSpec interface defines how WXS should monitor statistics.

Use the local StatsAccessor API to query data and access statistics on any ObjectGrid instance that is in the same Java virtual machine (JVM) as the running code. Use the following steps to enable monitoring of the internal statistics tree.

  1. Retrieve the StatsAccessor object. The StatsAccessor interface follows the singleton pattern. So, apart from problems related to the classloader, one StatsAccessor instance should exist for each JVM. This class serves as the main interface for all local statistics operations. The following code is an example of how to retrieve the accessor class. Call this operation before any other ObjectGrid calls. 
    public class LocalClient {

   public static void main(String[] args) {

      // retrieve a handle to the StatsAccessor StatsAccessor accessor = StatsAccessorFactory.getStatsAccessor();

   }}

  2. Set the data grid StatsSpec interface. Set this JVM to collect all statistics at the ObjectGrid level only. Ensure that an application enables all statistics that might be needed before you begin any transactions. The following example sets the StatsSpec interface using both a static constant field and using a spec String. Using a static constant field is simpler because the field has already defined the specification. However, by using a spec String, we can enable any combination of statistics that are required. 
    public static void main(String[] args) {

      // retrieve a handle to the StatsAccessor StatsAccessor accessor = StatsAccessorFactory.getStatsAccessor();

      // Set the spec via the static field StatsSpec spec = new StatsSpec(StatsSpec.OG_ALL);
      accessor.setStatsSpec(spec);

      // Set the spec via the spec String StatsSpec spec = new StatsSpec("og.all=enabled");
      accessor.setStatsSpec(spec);}

  3. Send transactions to the grid to force data to be collected for monitoring. To collect useful data for statistics, send transactions to the data grid. The following code excerpt inserts a record into MapA, which is in ObjectGridA. Because the statistics are at the ObjectGrid level, any map within the ObjectGrid yields the same results. 
    public static void main(String[] args) {

      // retrieve a handle to the StatsAccessor StatsAccessor accessor = StatsAccessorFactory.getStatsAccessor();

      // Set the spec via the static field StatsSpec spec = new StatsSpec(StatsSpec.OG_ALL);
      accessor.setStatsSpec(spec);

      ObjectGridManager manager = 
    ObjectGridmanagerFactory.getObjectGridManager();
      ObjectGrid grid = manager.getObjectGrid("ObjectGridA");
      Session session = grid.getSession();
      Map map = session.getMap("MapA");

      // Drive insert session.begin();
      map.insert("SomeKey", "SomeValue");
      session.commit();}

  4. Query a StatsFact using the StatsAccessor API. Every statistics path is associated with a StatsFact interface. The StatsFact interface is a generic placeholder used to organize and contain a StatsModule object. Before we can access the actual statistics module, the StatsFact object must be retrieved. 
    public static void main(String[] args)
{

      // retrieve a handle to the StatsAccessor StatsAccessor accessor = StatsAccessorFactory.getStatsAccessor();

      // Set the spec via the static field StatsSpec spec = new StatsSpec(StatsSpec.OG_ALL);
      accessor.setStatsSpec(spec);

      ObjectGridManager manager = 
    ObjectGridManagerFactory.getObjectGridManager();
      ObjectGrid grid = manager.getObjectGrid("ObjectGridA");
      Session session = grid.getSession();
      Map map = session.getMap("MapA");

      // Drive insert session.begin();
      map.insert("SomeKey", "SomeValue");
      session.commit();

      // Retrieve StatsFact 
      StatsFact fact = accessor.getStatsFact(new String[] {"EmployeeGrid"}, 
    StatsModule.MODULE_TYPE_OBJECT_GRID);}

  5. Interact with the StatsModule object. The StatsModule object is contained within the StatsFact interface. We can obtain a reference to the module using the StatsFact interface. Since the StatsFact interface is a generic interface, you must cast the returned module to the expected StatsModule type. Because this task collects eXtreme Scale statistics, the returned StatsModule object is cast to an OGStatsModule type. After the module is cast, you have access to all of the available statistics. 
    public static void main(String[] args) {

      // retrieve a handle to the StatsAccessor StatsAccessor accessor = StatsAccessorFactory.getStatsAccessor();

      // Set the spec via the static field StatsSpec spec = new StatsSpec(StatsSpec.OG_ALL);
      accessor.setStatsSpec(spec);

      ObjectGridManager manager = 
    ObjectGridmanagerFactory.getObjectGridManager();
      ObjectGrid grid = manager.getObjectGrid("ObjectGridA");
      Session session = grid.getSession();
      Map map = session.getMap("MapA");

      // Drive insert session.begin();
      map.insert("SomeKey", "SomeValue");
      session.commit();

      // Retrieve StatsFact StatsFact fact = accessor.getStatsFact(new String[] {"EmployeeGrid"}, 
    StatsModule.MODULE_TYPE_OBJECT_GRID);

      // Retrieve module and time OGStatsModule module = (OGStatsModule)fact.getStatsModule();
      ActiveTimeStatistic timeStat = 
    module.getTransactionTime("false ", true);
      double time = timeStat.getMeanTime(); }


Monitoring with the xscmd utility

The xscmd utility replaces the xsadmin sample utility as a fully supported monitoring and administration tool. With the xscmd utility, we can display textual information about your WXS topology.

For the xscmd utility to display results, you must have created the data grid topology. Your catalog servers and container servers must be started.

Use the xscmd utility to view the current layout and specific state of the data grid, such as map content. In this example, the layout of the data grid in this task consists of a single ObjectGridA data grid with one MapA map that belongs to the MapSetA map set. This example demonstrates how we can display all active containers within a data grid and print out filtered metrics regarding the map size of the MapA map. To see all possible command options, run the xscmd utility without any arguments or with the -help option.

  1. Monitor the environment with the xscmd utility.

    • To enable statistics for all of the servers, run the following command:

        ./xscmd.sh -c setStatsSpec -spec ALL=enabled -g ObjectGridA

    • To display all online container servers for a data grid, run the following command:

        ./xscmd.sh -c showPlacement -g ObjectGridA -ms MapSetA

      All container information is displayed.

      To obtain this information when Transport Layer Security/Secure Sockets Layer (TLS/SSL) is enabled, start the catalog and container servers with the JMX service port set. To set the JMX service port, we can either use the -JMXServicePort option on the startOgServer or startXsServer script or we can call the setJMXServicePort method on the ServerProperties interface.

    • To display information about the maps for the ObjectGridA data grid, run the following command:

        ./xscmd.sh -c showMapSizes -g ObjectGridA -ms MapSetA

    • To connect to the catalog service and display information about the MapA map for the entire catalog service domain, run the following command:

        ./xscmd.sh -c showMapSizes -g ObjectGridA -ms MapSetA -m MapA -cep CatalogMachine:6645

      The xscmd utility connects to the MBean server running on a catalog server. By connecting to a single catalog server, we can retrieve information about the entire catalog service domain. A catalog server can run as a stand-alone process, WebSphere Application Server process, or embedded within a custom application process. Use the -cep option to specify the catalog service host name and port. If you include a list of catalog servers for the -cep option, the catalog servers must be within the same catalog service domain. We can retrieve statistics for one catalog service domain at a time.

    • To display the configured and runtime placement of the configuration, run one of the following commands:

      • xscmd -c placementServiceStatus
      • xscmd -c placementServiceStatus -g ObjectGridA -ms MapSetA
      • xscmd -c placementServiceStatus -ms MapSetA
      • xscmd -c placementServiceStatus -g ObjectGridA

      We can scope the command to display placement information for the entire configuration, a single data grid, a single map set, or a combination of a data grid and map set.

  2. Display summaries of the replication states in the environment.

    • Display a summary of the outstanding revisions for each container server. We can run the command on a specific container server with the -ct argument, or all of the container servers if you do not include any arguments.

        ./xscmd.sh -c showReplicationState -ct container1

      The information in the output of this command includes outbound replication and inbound replication. Outbound replication includes the changes that must get pushed out from the primary shard on the container server to its replica shards on other container servers . Inbound replication includes changes that must get pushed from primary shards on other container servers to replicas on the container server. These statistics can give you an idea of the replication health. If the number of outstanding revisions on a container server suddenly grows very high, problems with the container might exist.

    • Display a summary of the outstanding revisions for shards between catalog service domains. We can run the command on a specific container server and catalog service domain, or your entire configuration if you do not include any arguments.

        ./xscmd.sh -c showDomainReplicationState -dom domainA -ct container1

      The information in the output of this command includes a summary of the outstanding revisions for each container server for each linked catalog service domain. The command returns the changes to be replicated between each primary shard and the corresponding remote primary shards that are in another catalog service domain.


Monitoring with WebSphere Application Server PMI

WebSphere eXtreme Scale supports Performance Monitoring Infrastructure (PMI) when running in a WebSphere Application Server or WebSphere Extended Deployment application server. PMI collects performance data on runtime applications and provides interfaces that support external applications to monitor performance data. Use the administrative console or the wsadmin tool to access monitoring data.

Use PMI to monitor your environment when we are using WebSphere eXtreme Scale combined with WebSphere Application Server.

WebSphere eXtreme Scale uses the custom PMI feature of WebSphere Application Server to add its own PMI instrumentation. With this approach, we can enable and disable WebSphere eXtreme Scale PMI with the administrative console or with Java Management Extensions (JMX) interfaces in the wsadmin tool. In addition, we can access WebSphere eXtreme Scale statistics with the standard PMI and JMX interfaces used by monitoring tools, including the Tivoli Performance Viewer.

  1. Enable eXtreme Scale PMI. You must enable PMI to view the PMI statistics.
  2. Retrieve eXtreme Scale PMI statistics. View the performance of your eXtreme Scale applications with the Tivoli Performance Viewer.


Enable PMI

Use WebSphere Application Server Performance Monitoring Infrastructure (PMI) to enable or disable statistics at any level. For example, we can choose to enable the map hit rate statistic for a particular map, but not the number of entry statistic or the loader batch update time statistic. We can enable PMI in the administrative console or with scripting.

Your application server must be started and have a WXS-enabled application installed. To enable PMI with scripting, you also must be able to log in and use the wsadmin tool.

Use WebSphere Application Server PMI to provide a granular mechanism with which we can enable or disable statistics at any level. For example, we can choose to enable the map hit rate statistic for a particular map, but not the number of entry or the loader batch update time statistics. This section shows how to use the administrative console and wsadmin scripts to enable ObjectGrid PMI.


Example

We can perform the following steps to enable PMI statistics for the sample application:

  1. Launch the application using the http://host:port/ObjectGridSample Web address, where host and port are the host name and HTTP port number of the server where the sample is installed.

  2. In the sample application, click ObjectGridCreationServlet, and then click action buttons 1, 2, 3, 4, and 5 to generate actions to the ObjectGrid and maps. Do not close this servlet page right now.

  3. In the administrative console...

      Monitoring and Tuning | Performance Monitoring Infrastructure | server_name | Runtime tab | Custom radio button

  4. Expand the ObjectGrid Maps module in the runtime tree, then click the clusterObjectGrid link. Under the ObjectGrid Maps group, there is an ObjectGrid instance called clusterObjectGrid, and under the clusterObjectGrid group four maps exist: counters, employees, offices, and sites. In the ObjectGrids instance, there is the clusterObjectGrid instance, and under that instance is a transaction type called DEFAULT.

  5. We can enable the statistics of interest to you. For example, we can enable number of map entries for employees map, and transaction response time for the DEFAULT transaction type.


What to do next

After PMI is enabled, we can view PMI statistics with the administrative console or through scripting.


Retrieve PMI statistics

By retrieving PMI statistics, we can see the performance of your eXtreme Scale applications.

We can retrieve PMI statistics to view in Tivoli Performance Viewer by completing steps in the administrative console or with scripting.


Results

We can view statistics in the Tivoli Performance Viewer.


PMI modules

We can monitor the performance of the applications with the performance monitoring infrastructure (PMI) modules.


objectGridModule

The objectGridModule contains a time statistic: transaction response time. A transaction is defined as the duration between the Session.begin method call and the Session.commit method call. This duration is tracked as the transaction response time. The root element of the objectGridModule, "root", serves as the entry point to the WebSphere eXtreme Scale statistics. This root element has ObjectGrids as its child elements, which have transaction types as their child elements. The response time statistic is associated with each transaction type.

The following diagram shows an example of the ObjectGridModule structure. In this example, two ObjectGrid instances exist in the system: ObjectGrid A and ObjectGrid B. The ObjectGrid A instance has two types of transactions: A and default. The ObjectGrid B instance has only the default transaction type.

Transaction types are defined by application developers because they know what types of transactions their applications use. The transaction type is set using the following Session.setTransactionType(String) method: 

/**
* Sets the transaction type for future transactions.
*
* After this method is called, all of the future transactions have the * same type until another transaction type is set. If  no transaction
* type is set, the default  TRANSACTION_TYPE_DEFAULT transaction type
* is used.
*
* Transaction types are used mainly for statistical data tracking purpose.
* Users can predefine types of transactions that run in an
* application. The idea is to categorize transactions with the same characteristics
* to one category (type), so one transaction response time statistic can be * used to track each transaction type.
*
* This tracking is useful when the application has different types of
* transactions.
* Among them, some types of transactions, such as update transactions, process
* longer than other transactions, such as read.only transactions. By using the * transaction type, different transactions are tracked by different statistics,
* so the statistics can be more useful.
*
* @param tranType the transaction type for future transactions.
*/
void setTransactionType(String tranType);
The following example sets transaction type to updatePrice: 
// Set the transaction type to updatePrice
// The time between session.begin() and session.commit() will be // tracked in the time statistic for "updatePrice".
session.setTransactionType("updatePrice");
session.begin();
map.update(stockId, new Integer(100));
session.commit();
The first line indicates that the subsequent transaction type is updatePrice. An updatePrice statistic exists under the ObjectGrid instance that corresponds to the session in the example. Using Java Management Extensions (JMX) interfaces, we can get the transaction response time for updatePrice transactions. We can also get the aggregated statistic for all types of transactions on the specified ObjectGrid instance.


mapModule

The mapModule contains three statistics that are related to WXS maps:

The root element of the mapModule, "root", serves as the entry point to the ObjectGrid Map statistics. This root element has ObjectGrids as its child elements, which have maps as their child elements. Every map instance has the three listed statistics. The mapModule structure is shown in the following diagram:

The following diagram shows an example of the mapModule structure:


hashIndexModule

The hashIndexModule contains the following statistics that are related to Map-level indexes:

The root element of the hashIndexModule, "root", serves as the entry point to the HashIndex statistics. This root element has ObjectGrids as its child elements, ObjectGrids have maps as their child elements, which finally have HashIndexes as their child elements and leaf nodes of the tree. Every HashIndex instance has the three listed statistics. The hashIndexModule structure is shown in the following diagram:

The following diagram shows an example of the hashIndexModule structure:


agentManagerModule

The agentManagerModule contains statistics that are related to map-level agents:

The root element of the agentManagerModule, "root", serves as the entry point to the AgentManager statistics. This root element has ObjectGrids as its child elements, ObjectGrids have maps as their child elements, which finally have AgentManager instances as their child elements and leaf nodes of the tree. Every AgentManager instance has statistics.

Figure 7. agentManagerModule structure

Figure 8. agentManagerModule structure example


queryModule

The queryModule contains statistics that are related to WXS queries:

The root element of the queryModule, "root", serves as the entry point to the Query Statistics. This root element has ObjectGrids as its child elements, which have Query objects as their child elements and leaf nodes of the tree. Every Query instance has the three listed statistics.

queryModule structure

QueryStats.jpg queryModule structure example


Access Managed Beans (MBeans) using the wsadmin tool

Use the wsadmin utility provided in WebSphere Application Server to access managed bean (MBean) information.

Run the wsadmin tool from the bin directory in your WebSphere Application Server installation. The following example retrieves a view of the current shard placement in a dynamic eXtreme Scale. We can run the wsadmin tool from any installation where eXtreme Scale is running. You do not have to run the wsadmin tool on the catalog service. 

$ wsadmin.sh -lang jython wsadmin>placementService = AdminControl.queryNames ("com.ibm.websphere.objectgrid:*,type=PlacementService")
wsadmin>print AdminControl.invoke(placementService,
 "listObjectGridPlacement","library ms1")

<objectGrid name="library" mapSetName="ms1">
  <container name="container-0" zoneName="false Domain" 
  hostName="host1.company.org" serverName="server1">
     <shard type="Primary" partitionName="0"/>
     <shard type="SynchronousReplica" partitionName="1"/>
  </container>
  <container name="container-1" zoneName="false Domain" 
  hostName="host2.company.org" serverName="server2">
     <shard type="SynchronousReplica" partitionName="0"/>
     <shard type="Primary" partitionName="1"/>
  </container>
  <container name="UNASSIGNED" zoneName="_ibm_SYSTEM" 
  hostName="UNASSIGNED" serverName="UNNAMED">
    <shard type="SynchronousReplica" partitionName="0"/>
    <shard type="AsynchronousReplica" partitionName="0"/>
  </container>
</objectGrid>


Monitoring server statistics with managed beans (MBeans)

Used managed beans (MBeans) to track statistics in your environment.

For the attributes to be recorded, you must enable statistics. We can enable statistics on the server, or enable HTTP session statistics to track attributes on your client application.

We can enable statistics in one of the following ways:


Example

  • Package com.ibm.websphere.objectgrid.management
  • Interface PlacementServiceMBean


    Monitoring client HTTP session statistics

    We can monitor user session activity for web applications running on your server.

    Enable HTTP session management in WebSphere eXtreme Scale.

    We can monitor HTTP session activity in the following types of installations:

    • Stand-alone installation in which WebSphere eXtreme Scale is using another application server
    • Integrated installation in which WebSphere eXtreme Scale is using WebSphere Application Server as the application server

    Depending on how you have deployed WebSphere eXtreme Scale, we can monitor different types of session activities:

    • When using WebSphere eXtreme Scale with another application server, we can monitor the following counters:

      Name Description
      createCount The number of sessions that were created.
      invalidateCount The number of sessions that were invalidated.
      activeCount The number of concurrently active sessions. A session is active if the WebSphere Application Server is currently processing a request that uses that session.
      liveCount The number of local sessions that are currently cached in memory from the time at which this metric is enabled
      cacheDiscardCount The number of session objects that have been forced out of the cache. A least recently used (LRU) algorithm removes old entries to make room for new sessions and cache misses. Applicable only for persistent sessions.
      affinityBreakCount The number of requests that are received for sessions that were last accessed from another Web application. This value can indicate failover processing or a corrupt plug-in configuration.
      timeoutInvalidationCount The number of sessions that are invalidated by timeout.
      activateNonExistSessionCount The number of requests for a session that no longer exists, presumably because the session timed out. Use this counter to help determine if the timeout is too short.

    • When using WebSphere Application Server, we can monitor the following counters: Servlet session counters .

    Depending on how you have deployed WebSphere eXtreme Scale we can enable HTTP client session statistics in one of the following ways:

    • If you have installed WebSphere eXtreme Scale in a stand-alone environment, then enable HTTP client session statistics with the splicer.properties file.

      • Modify the following statistics property from false to true: enableSessionStats=true.
      • Modify the following statistics property to all: sessionStatsSpec=session.all=enabled.

    • If you have installed WebSphere eXtreme Scale for session replication on WebSphere Application Server, then enable HTTP session monitoring activity with the Performance Monitoring Infrastructure (PMI) service in WebSphere Application Server. For more information, see Enabling PMI using the administrative console .

      Even if you plan to use the PMI console to monitor session activity in WebSphere Application Server, we can also enable HTTP session activity in WebSphere eXtreme Scale to ensure we are able to monitor the session statistic types in both products.


    What to do next

    After you enable HTTP session statistics in WebSphere eXtreme Scale, we can view the statistics through the registered MBean: com.ibm.websphere.objectgrid:type=Session,name=webAppContextRoot

    We can view the MBean using the one of the following tools:

    • Access MBean statistics using the wsadmin tool.
    • Access MBean statistics programmatically.
    • Access MBean statistics with tools such as JConsole (Java Monitoring and Management Console).


    Monitoring with vendor tools

    WebSphere eXtreme Scale can be monitored using several popular enterprise monitoring solutions. Plug-in agents are included for IBM Tivoli Monitoring and Hyperic HQ, which monitor WebSphere eXtreme Scale using publicly accessible management beans. CA Wily Introscope uses Java method instrumentation to capture statistics.

    Configure WebSphere Commerce to use WebSphere eXtreme Scale for dynamic cache to improve performance and scale

    WebSphere Business Process Management and Connectivity integration


    Monitoring with the IBM Tivoli Enterprise Monitoring Agent for WebSphere eXtreme Scale

    The IBM Tivoli Enterprise Monitoring Agent is a feature-rich monitoring solution that we can use to monitor databases, operating systems and servers in distributed and host environments. WebSphere eXtreme Scale includes a customized agent that we can use to introspect eXtreme Scale management beans. This solution works effectively for both stand-alone eXtreme Scale and WebSphere Application Server deployments.

    • Install WebSphere eXtreme Scale Version 7.0.0 or later.

      Also, statistics must be enabled to collect statistical data from WebSphere eXtreme Scale servers.

    • Install IBM Tivoli Monitoring Version 6.2.1 with fix pack 2 or later.

    • Install the Tivoli OS agent on each server or host on which eXtreme Scale servers run.

    • Install the WebSphere eXtreme Scale agent, which we can download for free from the IBM Open Process Automation Library (OPAL) site .

    Complete the following steps to install and configure the Tivoli Monitoring Agent:

    1. Install the Tivoli Monitoring Agent for WebSphere eXtreme Scale.

      Download the Tivoli installation image and extract its files to a temporary directory.

    2. Install eXtreme Scale application support files.

      Install eXtreme Scale application support on each of the following deployments.

      • Tivoli Enterprise Portal Server (TEPS)
      • Enterprise Desktop client (TEPD)
      • Tivoli Enterprise Monitoring Server (TEMS)

      1. From the temporary directory that you created, start a new command window and run the appropriate executable file for your platform. The installation script automatically detects your Tivoli deployment type (TEMS, TEPD, or TEPS). We can install any type on a single host or on multiple hosts; and all of the three deployment types require the installation of the eXtreme Scale agent application support files.

      2. In the Installer window, verify that the selections for the Tivoli Components deployed are correct. Click Next.

      3. If we are prompted, submit your hostname and administrative credentials. Click Next.

      4. Select the Monitoring Agent for WebSphere eXtreme Scale. Click Next.

      5. You are notified of what installation actions are to be performed. Click Next, and we can see the progress of the installation until completion.

      After completing the procedure, all application support files required by WebSphere eXtreme Scale agent are installed.

    3. Install the agent on each of the eXtreme Scale nodes.

      You install a Tivoli OS agent on each of the computers. You do not need to configure or start this agent. Use the same installation image from the previous step to run the platform specific executable file.

      As a guideline, you need to install only one agent per host. Each agent is capable of supporting many instances of eXtreme Scale servers. For best performance, use one agent instance for monitoring about 50 eXtreme Scale servers.

      1. From the installation wizard welcome screen, click Next to open the screen to specify installation path information.

      2. For the Tivoli Monitoring installation directory field, enter or browse to...

          /opt/IBM/ITM

        For the Location for installable media field, verify that the displayed value is correct and click Next.

      3. Select the components we want to add, such as Perform a local install of the solution and click Next.

      4. Select the applications for which to add support for by selecting the application, such as Monitoring Agent for WebSphere eXtreme Scale, and click Next.

      5. We can see the progress until application support is added successfully.

      Repeat these steps on each of the eXtreme Scale nodes. We can also use silent installation. See the IBM Tivoli Monitoring Information Center for more information about silent installation.

    4. Configure the WebSphere eXtreme Scale agent.

      Each of the agents installed need to be configured to monitor any catalog server, eXtreme Scale server, or both.

      The steps to configure Windows and UNIX platforms are different. Configuration for the Windows platform is completed with the Manage Tivoli Monitoring Services user interface. Configuration for UNIX platforms is command-line based.

      Use the following steps to initially configure the agent on Windows

      1. From the Manage Tivoli Enterprise Monitoring Services window...

          Start | All Programs | IBM Tivoli Monitoring | Manage Tivoli Monitoring Services

      2. Right click on Monitoring Agent for WebSphere eXtreme Scale and select Configure using default, which opens a window to create a unique instance of the agent.

      3. Choose a unique name: for example, instance1, and click Next.

      • If you plan to monitor stand-alone eXtreme Scale servers, complete the following steps:

        1. Update the Java parameters, ensure that the Java Home value is correct. JVM arguments can be left empty. Click Next.

        2. Select the type of MBean server connection type, Use

          JSR-160-Complaint Server for stand-alone eXtreme Scale servers. Click Next.

        3. If security is enabled, update User ID and Password values. Leave the JMX service URL value as is. You override this value later. Leave the JMX Class Path Information field as it is. Click Next.

        To configure the servers for the agent on Windows, complete the following steps:

        1. Set up subnode instances of eXtreme Scale servers in the WebSphere eXtreme Scale Grid Servers pane. If no container servers exist on your computer, click Next to proceed to the catalog service pane.

        2. If multiple eXtreme Scale container servers exist on your computer, configure the agent to monitor each one server.

        3. We can add as many eXtreme Scale servers as you require, if their names and ports are unique, by clicking New. (When a WXS server is started, a JMXPort value must be specified.)

        4. After you configure the container servers, click Next, which brings you to the WebSphere eXtreme Scale Catalog Servers pane.

        5. If you have no catalog servers, click OK. If you have catalog servers, add a new configuration for each server, as you did with the container servers . Again, choose a unique name, preferably the same name used when starting the catalog service. Click OK to finish.

      • If you plan to monitor servers for the agent on eXtreme Scale servers that are embedded within a WebSphere Application Server process, complete the following steps:

        1. Update the Java parameters, ensure that the Java Home value is correct. JVM arguments can be left empty. Click Next.

        2. Select the MBean server connection type. Select the WebSphere Application Server version that is appropriate for your environment. Click Next.

        3. Ensure that the WebSphere Application Server information in the panel is correct. Click Next.

        4. Add only one subnode definition. Give the subnode definition a name, but do not update the port definition. Within WebSphere Application Server environment, data can be collected from all the application server that are managed by the node agent running on the computer. Click Next.

        5. If there no catalog servers exist in the environment, click OK. If you have catalog servers, add a new configuration for each catalog server, similarly to the container servers . Choose a unique name for the catalog service, preferably the same name that you use when starting the catalog service. Click OK to finish.

      The container servers do not need to be collocated with the catalog service.

      Now that the agent and servers are configured and ready, on the next window, right click on

      instance1 to start the agent.

      To configure the agent on the UNIX platform on the command line, complete the following steps:

      An example follows for stand-alone servers that uses a JSR160 Compliant connection type. The example shows three eXtreme Scale containers on the single host (rhea00b02) and the JMX listener addresses are 15000,15001 and 15002 respectively. There are no catalog servers.

      Output from the configuration utility displays in monospace italics, while the user response is in monospace bold. (If no user response was required, the default was selected by pressing the enter key.) 

      rhea00b02 # ./itmcmd config -A xt
Agent configuration started...
Enter instance name (default is: ): inst1
Edit "Monitoring Agent for WebSphere eXtreme Scale" settings? [ 1=Yes, 2=No ] (default is: 1):
Edit 'Java' settings? [ 1=Yes, 2=No ] (default is: 1):
Java home (default is: C:\Program Files\IBM\Java50): /opt/OG61/java
Java trace level [ 1=Error, 2=Warning, 3=Information, 4=Minimum Debug, 5=Medium Debug, 6=Maximum Debug,
    7=All ] (default is: 1):
JVM arguments (default is: ):
Edit 'Connection' settings? [ 1=Yes, 2=No ] (default is: 1):
MBean server connection type [ 1=JSR-160-Compliant Server, 2=WebSphere Application Server version 6.0, 
3=WebSphere Application Server version 6.1, 4=WebSphere Application Server version 7.0 ] (default is: 1): 1
Edit 'JSR-160-Compliant Server' settings? [ 1=Yes, 2=No ] (default is: 1):
JMX user ID (default is: ):
Enter JMX password (default is: ):          
Re-type : JMX password (default is: ):
JMX service URL (default is: service:jmx:rmi:///jndi/rmi://localhost:port/objectgrid/MBeanServer):
----------------------------------------
JMX Class Path Information
JMX base paths (default is: ):
JMX class path (default is: ):
JMX JAR directories (default is: ):
Edit 'WebSphere eXtreme Scale Catalog Service' settings? [ 1=Yes, 2=No ] (default is: 1): 2
Edit 'WebSphere eXtreme Scale Grid Servers' settings? [ 1=Yes, 2=No ] (default is: 1): 1
No 'WebSphere eXtreme Scale Grid Servers' settings available?
Edit 'WebSphere eXtreme Scale Grid Servers' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 4): 1
WebSphere eXtreme Scale Grid Servers (default is: ): rhea00b02_c0
JMX service URL (default is: service:jmx:rmi:///jndi/rmi://localhost:<port>/objectgrid/MBeanServer): 
service:jmx:rmi:///jndi/rmi://localhost:15000/objectgrid/MBeanServer

'WebSphere eXtreme Scale Grid Servers' settings:  WebSphere eXtreme Scale Grid Servers=ogx
Edit 'WebSphere eXtreme Scale Grid Servers' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 4): 1
WebSphere eXtreme Scale Grid Servers (default is: ): rhea00b02_c1
JMX service URL (default is: service:jmx:rmi:///jndi/rmi://localhost:<port>/objectgrid/MBeanServer):
service:jmx:rmi:///jndi/rmi://localhost:15001/objectgrid/MBeanServer

'WebSphere eXtreme Scale Grid Servers' settings:  WebSphere eXtreme Scale Grid Servers= rhea00b02_c1
Edit 'WebSphere eXtreme Scale Grid Servers' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 4): 1
WebSphere eXtreme Scale Grid Servers (default is: ): rhea00b02_c2
JMX service URL (default is: service:jmx:rmi:///jndi/rmi://localhost:<port>/objectgrid/MBeanServer): 
service:jmx:rmi:///jndi/rmi://localhost:15002/objectgrid/MBeanServer

'WebSphere eXtreme Scale Grid Servers' settings:  WebSphere eXtreme Scale Grid Servers= rhea00b02_c2
Edit 'WebSphere eXtreme Scale Grid Servers' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 4): 5

Will this agent connect to a TEMS? [1=YES, 2=NO] (Default  is: 1):
TEMS Host Name (Default  is: rhea00b00):

Network Protocol [ip, sna, ip.pipe or ip.spipe] (Default  is: ip.pipe):

     Now choose the next protocol number from one of these:
     - ip
     - sna
     - ip.spipe
     - 0 for none
Network Protocol 2 (Default  is: 0):
IP.PIPE Port Number (Default  is: 1918):
Enter name of KDC_PARTITION (Default  is: null):

Configure connection for a secondary TEMS? [1=YES, 2=NO] (Default  is: 2):
Enter Optional Primary Network Name or 0 for "none" (Default  is: 0):
Agent configuration completed...

      The previous example creates an agent instance called .inst1., and updates the Java Home settings. The eXtreme Scale container servers are configured, but the catalog service is not configured. The previous procedure creates a text file of the following format in the directory: <ITM_install>/config/<host>_xt_<instance name>.cfg.

      Example: rhea00b02_xt_inst1.cfg

      It is best to edit this file with your choice of plain text editor. An example of the content of such the file follows: 

      INSTANCE=inst2 [SECTION=KQZ_JAVA [ { JAVA_HOME=/opt/OG61/java }  { JAVA_TRACE_LEVEL=ERROR }  ]
SECTION=KQZ_JMX_CONNECTION_SECTION [ { KQZ_JMX_CONNECTION_PROPERTY=KQZ_JMX_JSR160_JSR160 }  ]
SECTION=KQZ_JMX_JSR160_JSR160 [ { KQZ_JMX_JSR160_JSR160_CLASS_PATH_TITLE= }  
{ KQZ_JMX_JSR160_JSR160_SERVICE_URL=service:jmx:rmi:///jndi/rmi://localho
st:port/objectgrid/MBeanServer }  { KQZ_JMX_JSR160_JSR160_CLASS_PATH_SEPARATOR= }  ]
SECTION=OGS:rhea00b02_c1 [ { KQZ_JMX_JSR160_JSR160_SERVICE_URL=service:jmx:
rmi:///jndi/rmi://localhost:15001/objectgrid/MBeanServer }  ]
SECTION=OGS:rhea00b02_c0 [ { KQZ_JMX_JSR160_JSR160_SERVICE_URL=service:jmx:
rmi:///jndi/rmi://localhost:15002/objectgrid/MBeanServer }  ]
SECTION=OGS:rhea00b02_c2 [ { KQZ_JMX_JSR160_JSR160_SERVICE_URL=service:jmx:
rmi:///jndi/rmi://localhost:15002/objectgrid/MBeanServer }  ]]

      An example that shows a configuration on a WebSphere Application Server deployment follows: 

      rhea00b02 # ./itmcmd config -A xt
Agent configuration started...
Enter instance name (default is: ): inst1
Edit "Monitoring Agent for WebSphere eXtreme Scale" settings? [ 1=Yes, 2=No ] (default is: 1): 1
Edit 'Java' settings? [ 1=Yes, 2=No ] (default is: 1): 1
Java home (default is: C:\Program Files\IBM\Java50): /opt/WAS61/java
Java trace level [ 1=Error, 2=Warning, 3=Information, 4=Minimum Debug, 5=Medium Debug, 6=Maximum Debug,
    7=All ] (default is: 1):
JVM arguments (default is: ):
Edit 'Connection' settings? [ 1=Yes, 2=No ] (default is: 1):
MBean server connection type [ 1=JSR-160-Compliant Server, 2=WebSphere Application Server version 6.0, 
3=WebSphere Application Server version 6.1, 4=WebSphere Application Server version 7.0 ] (default is: 1): 4
Edit 'WebSphere Application Server version 7.0' settings? [ 1=Yes, 2=No ] (default is: 1):WAS user ID (default is: ):
Enter WAS password (default is: ):     
Re-type : WAS password (default is: ):
WAS host name (default is: localhost): rhea00b02
WAS port (default is: 2809):
WAS connector protocol [ 1=rmi, 2=soap ] (default is: 1):
WAS profile name (default is: ): default ----------------------------------------
WAS Class Path Information
WAS base paths (default is: C:\Program Files\IBM\WebSphere\AppServer;/opt/IBM/WebSphere/AppServer): /opt/WAS61
WAS class path (default is: runtimes/com.ibm.ws.admin.client_6.1.0.jar;runtimes/com.ibm.ws.ejb.thinclient_7.0.0.jar):
WAS JAR directories (default is: lib;plugins):
Edit 'WebSphere eXtreme Scale Grid Servers' settings? [ 1=Yes, 2=No ] (default is: 1):
No 'WebSphere eXtreme Scale Grid Servers' settings available?
Edit 'WebSphere eXtreme Scale Grid Servers' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 4): 1
WebSphere eXtreme Scale Grid Servers (default is: ): rhea00b02
JMX service URL (default is: service:jmx:rmi:///jndi/rmi://localhost:<port>/objectgrid/MBeanServer):

'WebSphere eXtreme Scale Grid Servers' settings:  WebSphere eXtreme Scale Grid Servers=rhea00b02
Edit 'WebSphere eXtreme Scale Grid Servers' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 4): 5
Edit 'WebSphere eXtreme Scale Catalog Service' settings? [ 1=Yes, 2=No ] (default is: 1): 2
Will this agent connect to a TEMS? [1=YES, 2=NO] (Default  is: 1):
TEMS Host Name (Default  is: rhea00b02):

Network Protocol [ip, sna, ip.pipe or ip.spipe] (Default  is: ip.pipe):

     Now choose the next protocol number from one of these:
     - ip
     - sna
     - ip.spipe
     - 0 for none
Network Protocol 2 (Default  is: 0):
IP.PIPE Port Number (Default  is: 1918):
Enter name of KDC_PARTITION (Default  is: null):

Configure connection for a secondary TEMS? [1=YES, 2=NO] (Default  is: 2):
Enter Optional Primary Network Name or 0 for "none" (Default  is: 0):
Agent configuration completed...
rhea00b02 #
      For WebSphere Application Server deployments, you do not need to create multiple sub nodes. The eXtreme Scale agent connects to the node agent to gather all the information from application servers for which it is responsible.

      SECTION=CAT signifies a catalog service line whereas SECTION=OGS signifies a WXS server configuration line.

    5. Configure the JMX port for all eXtreme Scalecontainer servers.

      When eXtreme Scale container servers are started, without specifying the -JMXServicePort argument, an MBean server is assigned a dynamic port. The agent needs to know in advance with which JMX port to communicate. The agent does not work with dynamic ports.

      When you start the servers, specify the -JMXServicePort <port_number> argument when you start the eXtreme Scale server using the start server command. Running this command ensures that the JMX server within the process listens to a static pre-defined port.

      For the previous examples in a UNIX installation, two eXtreme Scale servers need to be started with ports set:

      1. "-JMXServicePort" "15000" (for rhea00b02_c0)

      2. "-JMXServicePort" "15001" (for rhea00b02_c1)

      1. Start the eXtreme Scale agent.

        Assuming the inst1 instance was created, as in the previous example, issue the following commands.

        1. cd <ITM_install>/bin

        2. itmcmd agent .o inst1 start xt

      2. Stop the eXtreme Scale agent.

        Assuming .inst1. was the instance created, as in the previous example, issue the following commands.

        1. cd <ITM_install>/bin

        2. itmcmd agent .o inst1 stop xt

    6. Enable Statistics for all eXtreme Scale container servers .

      The agent uses the eXtreme Scale statistics MBeans to record statistics. The eXtreme Scale statistics specification must be enabled using one of the following methods.


    Results

    After all servers are configured and started, MBeans data is displayed on the IBM Tivoli Portal console. Predefined workspaces show graphs and data metrics at each node level.

    The following workspaces are defined: eXtreme Scale Grid Servers node for all nodes monitored.

    • eXtreme Scale Transactions View
    • eXtreme Scale Primary Shard View
    • eXtreme Scale Memory View
    • eXtreme Scale ObjectMap View

    We can also configure your own workspace. For more information, see the information about customizing workspaces in the IBM Tivoli Monitoring Information Center .


    Monitoring eXtreme Scale applications with CA Wily Introscope

    CA Wily Introscope is a third-party management product that we can use to detect and diagnose performance problems in enterprise application environments. eXtreme Scale includes details on configuring CA Wily Introscope to introspect select portions of the eXtreme Scale run time to quickly view and validate eXtreme Scale applications. CA Wily Introscope works effectively for both stand-alone and WebSphere Application Server deployments.


    Overview

    To monitor eXtreme Scale applications with CA Wily Introscope, you must put settings into the ProbeBuilderDirective (PBD) files that give you access to the monitoring information for eXtreme Scale.

    The instrumentation points for Introscope might change with each fix pack or release. When you install a new fix pack or release, check the documentation for any changes in the instrumentation points. We can configure CA Wily Introscope ProbeBuilderDirective (PBD) files to monitor your eXtreme Scale applications. CA Wily Introscope is an application management product with which we can proactively detect, triage and diagnose performance problems in your complex, composite and Web application environments.


    PBD file settings for monitoring the catalog service

    Use one or more of the following settings in your PBD file to monitor the catalog service. 

    TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl changeDefinedCompleted 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl viewChangeCompleted 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl viewAboutToChange 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeat 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatCluster 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatCurrentLeader 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatDeadServer 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatNewLeader 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatNewServer 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.PlacementServiceImpl 
importRouteInfo BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.PlacementServiceImpl heartbeat 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.PlacementServiceImpl joinPlacementGroup 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" TraceOneMethodOfClass: 
com.ibm.ws.objectgrid.catalog.placement.PlacementServiceImpl classifyServer 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.BalanceGridEventListener shardActivated 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.BalanceGridEventListener shardDeactivate 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}"

    Classes for monitoring the catalog service

    HAControllerImpl

    The HAControllerImpl class handles core group life cycle and feedback events. We can monitor this class to get an indication of the core group structure and changes.

    ServerAgent

    The ServerAgent class is responsible for communicating core group events with the catalog service. We can monitor the various heartbeat calls to spot major events.

    PlacementServiceImpl

    The PlacementServiceImpl class coordinates the containers. Use the methods on this class to monitor server join and placement events.

    BalanceGridEventListener

    The BalanceGridEventListener class controls the catalog leadership. We can monitor this class to get an indication of which catalog service is currently acting as the leader.


    PBD file settings for monitoring the containers

    Use one or more of the following settings in your PBD file to monitor the containers. 

    TraceOneMethodOfClass: com.ibm.ws.objectgrid.ShardImpl processMessage 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.plugins.CommittedLogSequenceListenerProxy applyCommitted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.plugins.CommittedLogSequenceListenerProxy sendApplyCommitted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.map.BaseMap evictMapEntries 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.checkpoint.CheckpointMapImpl$CheckpointIterator activateListener 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl changeDefinedCompleted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl viewChangeCompleted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl viewAboutToChange 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent batchProcess 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeat 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatCluster 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatCurrentLeader 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatDeadServer 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatNewLeader 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent heartbeatNewServer 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}"

    Classes for monitoring the containers

    ShardImpl

    The ShardImpl class has the processMessage method. The processMessage method is the method for client requests. With this method, we can get server side response time and request counts. By watching the counts across all the servers and monitoring heap utilization, we can determine if the grid is balanced.

    CheckpointIterator

    The CheckpointIterator class has the activateListener method call which puts primaries into peer mode. When the primaries are put into peer mode, the replica is up to date with the primary after the method completes. When a replica is regenerating from a full primary, this operation can take an extended period of time. The system is not fully recovered until this operation completes, so we can use this class to monitor the progress of the operation.

    CommittedLogSequenceListenerProxy

    The CommittedLogSequenceListenerProxy class has two methods of interest. The applyCommitted method runs for every transaction and the sendApplyCommitted runs as the replica is pulling information. The ratio of how often these two methods run can give you some indication of how well the replica is able to keep up with the primary.


    PBD file settings for monitoring the clients

    You can use one or more of the following settings in your PBD file to monitor the clients. 

    TraceOneMethodOfClass: com.ibm.ws.objectgrid.client.ORBClientCoreMessageHandler sendMessage 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.corba.cluster.ClusterStore bootstrap 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.corba.cluster.ClusterStore epochChangeBootstrap 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.map.BaseMap evictMapEntries 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.cluster.orb.routing.SelectionServiceImpl routeFailed 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.cluster.orb.routing.SelectionServiceImpl routeFailed 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.SessionImpl getMap 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.ObjectGridImpl getSession 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TurnOn: ObjectMap 
SetFlag: ObjectMap 
IdentifyClassAs: com.ibm.ws.objectgrid.ObjectMapImpl ObjectMap 
TraceComplexMethodsifFlagged: ObjectMap BlamePointTracerDifferentMethods 
"OGclient|{classname}|{method}"

    Classes for monitoring the clients

    ORBClientCoreMessageHandler

    The ORBClientCoreMessageHandler class is responsible for sending application requests to the containers. We can monitor the sendMessage method for client response time and number of requests.

    ClusterStore

    The ClusterStore class holds the routing information on the client side.

    BaseMap

    The BaseMap class has the evictMapEntries method that is called when the evictor wants to remove entries from the map.

    SelectionServiceImpl

    The SelectionServiceImpl class makes the routing decisions. If the client is making failover decisions, we can use this class to see the actions that are completed from the decisions.

    ObjectGridImpl

    The ObjectGridImpl class has the getSession method that we can monitor to see the number of requests to this method.


    Monitoring eXtreme Scale with Hyperic HQ

    Hyperic HQ is a third-party monitoring solution that is available freely as an open source solution or as an enterprise product. WebSphere eXtreme Scale includes a plug-in that allows Hyperic HQ agents to discover eXtreme Scale container servers and to report and aggregate statistics using eXtreme Scale management beans. Use Hyperic HQ to monitor stand-alone eXtreme Scale deployments.

    • This set of instructions is for Hyperic Version 4.0. If you have a newer version of Hyperic, see the Hyperic documentation for information such as the path names and how to start agents and servers.

    • Download the Hyperic server and agent installations. One server installation must be running. To detect all of the eXtreme Scale servers, a Hyperic agent must be running on each machine on which a WXS server is running. See the Hyperic website for download information and documentation support.

    • You must have access to the objectgrid-plugin.xml and hqplugin.jar files. These files are in...

        /path/to/hyperic/etc

    By integrating eXtreme Scale with Hyperic HQ monitoring software, we can graphically monitor and display metrics about the performance of your environment. You set up this integration by using a plug-in implementation on each agent.

    1. Start your eXtreme Scale servers. The Hyperic plug-in looks at the local processes to attach to the Java virtual machines running eXtreme Scale. To properly attach to the JVMs, each server must be started with the -jmxServicePort option.

    2. Put the extremescale-plugin.xml file and the wxshyperic.jar file in the appropriate server and agent plug-in directories in your Hyperic configuration. To integrate with Hyperic, both the agent and server installations must have access to the plug-in and JAR files. Although the server can dynamically swap configurations, you should complete the integration before you start any of the agents.

      1. Place the extremescale-plugin.xml file in the server plugin directory, which is at the following location: 
        hyperic_home/server_home/hq-engine/server/default/deploy/hq.ear/hq-plugins

      2. Place the extremescale-plugin.xml file in the agent plugin directory, which is at the following location: 
        agent_home/bundles/gent-4.0.2-939/pdk/plugins

      3. Put the wshyperic.jar file in the agent lib directory, which is at the following location 
        agent_home/bundles/gent-4.0.2-939/pdk/lib

    3. Configure the agent. The agent.properties file serves as a configuration point for the agent runtime. This property is in the agent_home/conf directory. The following keys are optional, but of importance to the eXtreme Scale plug-in:

      • autoinventory.defaultScan.interval.millis=<time_in_milliseconds>

        Set the interval in milliseconds between Agent discoveries.

      • log4j.logger.org.hyperic.hq.plugin.extremescale.XSServerDetector=DEBUG

        Enables verbose debug statements from the eXtreme Scale plug-in.

      • username=<username>

        Set the JMX user name if security is enabled.

      • password=<password>

        Set the JMX password if security is enabled.

      • sslEnabled=<true|false>

        Tells the plug-in whether or not to use SSL. The value is false by default.

      • trustPath=<path>

        Set the trust path for the SSL connection.

      • trustType=<type>

        Set the trust type for the SSL connection.

      • trustPass=<password>

        Set the trust password for the SSL connection.

    4. Start the agent discovery. The Hyperic agents send discoveries and metrics information to the server. Use the server to customize data views and group logical inventory objects to generate useful information. After the server is available, run the launch script or start the Windows service for the agent:

      • agent_home/bin/hq-agent.sh start

      • Start the agent with the Windows service.

      After you start the agents, the servers are detected and groups are configured. We can log into the server console and choose which resources to add to the inventory database for the server. The server console is at the following URL by default: http://<server_host_name>:7080/

    5. Statistics must be enabled for Hyperic to collect statistical data.

      Use the SetStatsSpec control action on the Hyperic console for eXtreme Scale. Navigate to the resource, then use the Control Action drop-down list on the Control tabbed page to specify a SetStatsSpec setting with ALL=enabled in the Control Arguments text box.

      Catalog servers are not detected by the filter set in the Hyperic console. See the information about the statsSpec property in Server properties file, which enable statistics as soon as the containers start.

    6. Monitor servers with the Hyperic console. After the servers are added to the inventory model, their services are no longer needed.

      • Dashboard view: When you viewed the resource detection events, you logged into the main dashboard view. The dashboard view is a generic view that acts as a message center that we can customize. We can export graphs or inventory objects to this main dashboard.

      • Resources view: We can query and view the entire inventory model from this page. After the services have been added, we can see every eXtreme Scale server properly labeled and listed together under the servers section. We can click on the individual servers to see the basic metrics.

    7. View the entire server inventory on the Resource View page. On this page, we can then select multiple ObjectGrid servers and group them together. After you group a set of resources, their common metrics can be graphed to show overlays and differences among group members. To display an overlay, select the metrics on the display of your Server Group. The metric then displays in the charting area. To display an overlay for all group members, click the underlined metric name. We can export any of the charts, node views, and comparative overlays to the main dashboard with the Tools menu.


    11. Monitoring eXtreme Scale information in DB2

    When the JPALoader or JPAEntityLoader is used with DB2 as the back-end database, eXtreme Scale-specific information can be passed to DB2. We can view this information by a performance monitor tool such as DB2 Performance Expert to monitor the eXtreme Scale applications that are accessing the database.

    When the loader is configured to use DB2 as the back-end database, the following eXtreme Scale information can be passed to DB2 for monitoring purposes:

    • User: Name of the user that authenticates to WXS. When basic authentication is not used, the principals from the authentication are used.

    • Workstation Name: Host name, IP of the eXtreme Scale container server.

    • Application Name: Name of the ObjectGrid, Persistence Unit name (if set).

    • Accounting Information: Specifies the thread ID, transaction type, transaction id, and the connection string.

    Read about the DB2 Performance Expert to learn how to monitor database access.

    • To enable all eXtreme Scale client information, set the following trace strings: 
      ObjectGridClientInfo*=event=enabled

    • To enable all but user information, use one of the following settings: 

      • ObjectGridClientInfo*=event=enabled,ObjectGridClientInfoUser=event=disabled

        or 

      • ObjectGridClientInfo=event=enabled


    Results

    After you turn on the trace function, data displays in the performance monitor tool such as DB2 Performance Expert.


    Example

    In the following example, user

    bob is authenticated as a WXS user. The application is accessing the mygrid data grid using the DB2Hibernate persistence unit. The container server is named

    XS_Server1. The resulting information follows:

    • User=bob

    • Workstation Name=XS_Server1,192.168.1.101

    • Application Name=mygrid,DB2Hibernate

    • Accounting Information=1, DEFAULT,FE7954BD-0126-4000-E000-2298094151DB,com.ibm.db2.jcc.t4.b@71787178

    In the following example, user

    bob is authenticated using a WebSphere Application Server token. The application is accessing the mygrid data grid using the DB2OpenJPA persistence unit name. The container server is named

    XS_Server2. The resulting information follows:

    • User 
      =acme.principal.UserPrincipal[Bob],acme.principal.
GroupPrincipal[admin]

    • Workstation Name=XS_Server2,192.168.1.102

    • Application Name=mygrid,DB2OpenJPA

    • Accounting Information=188,DEFAULT,FE72BC63-0126-4000-E000-851C092A4E33,com.ibm.ws.rsadapter.jdbc.WSJccSQLJConnection@2b432b43