IBM Tivoli Monitoring > Version 6.3 Fix Pack 2 > Administrator's Guide > Manage historical data > Use your historical data for analytic usage

IBM Tivoli Monitoring, Version 6.3 Fix Pack 2


Warehouse Proxy agent for analytics

You can enable the Warehouse Proxy agent to write incoming historical data to the Tivoli Data Warehouse and to a comma-separated (CSV) file. Best practice is to only write data to a CSV file if the data will be consumed by an external application, such as an analytic engine. You must have a connection to the Tivoli Data Warehouse to create CSV files.

The Warehouse Proxy agent receives export requests from agents that are stored in a memory queue where they can sit for a long time depending on how busy the Warehouse Proxy agent is. This wait can affect analytics that rely on the data from the Tivoli Data Warehouse because of a latency between the data collection at the agent, the data being exported to the Warehouse Proxy agent, and the actual write to the Tivoli Data Warehouse database.

For quicker access to data, you can now configure the Warehouse Proxy agent to write to a local CSV file as the data from the agent is received, allowing faster processing and results in a reduction of the load on the Tivoli Data Warehouse.

Before creating CSV files, ensure the following conditions are met:

Automatic maintenance is not available for the CSV files created. However, the total size of the written files is monitored, and file creation stops if the total size exceeds a configured threshold. The evaluation period and maximum size value are configurable environment variables.

No provision exists for avoiding duplication of data. The data sent might fail to be inserted into the Tivoli Data Warehouse due to any number of reasons. Events such as a timeout or database failure can cause the data to be resent. The Warehouse Proxy agent has no internal state or knowledge that the data is a resend, because the new data might have more rows or data added to it since data collection has occurred on the agent.


File format

The CSV file is created as soon as the data is completely exported by the agent. Initially, the file is created with a temporary name, and when all the data has been written the file is named as follows: <TABLENAME>_<MIN TIMESTAMP>_<MAX TIMESTAMP>_<SEQUENCE NUMBER>.csv. Data is written to the CSV files in UTF8 format.

You can use the KHD_CSV_ISO_DATE_FORMAT variable to change the default proprietary timestamp in the filename to ISO format. For more information see the "Parameters for configuration" section in this topic.

The data is written in the export order as it is sent by the agent. The data includes a header line containing the column names for readability purposes, and to provide descriptive information to the data.

For example, a file named WTLOGCLDSK_110314090000000_1110314090000001_00001.csv might contain the following information:

The columns in the CSV file might be a subset of the columns in the Tivoli Data Warehouse, depending on the level of the agent sending the data.


Parameters for configuration

Warehouse Proxy agent environment variables

Use the following environment variables to configure your Warehouse Proxy agent:

KHD_CSV_EVAL_INTERVAL

Optional. The interval in seconds between evaluation of the total size of the CSV files in the output directory. The time is specified in seconds, with a minimum of 60 seconds (1 minute). The default is 900 seconds (15 minutes).

KHD_CSV_ISO_DATE_FORMAT

Optional. Use Y to enable the conversion of proprietary format timestamps to ISO timestamps.

A proprietary format timestamp is defined as a fixed string of 16 characters in the format CYYMMDDHHMMSSmmm, where

  • C = Century (0 for 20th, 1 for 21st)

  • Y = Year

  • M = Month

  • D = Day

  • H = Hour

  • M = Minute

  • S = Second

  • m = Millisecond

For example:

  • Proprietary timestamp: 1130513094020000

  • ISO timestamp: 2013-05-13T09:40:20.000

If the column is declared as a timestamp of 16 characters but does not meet the requirements as described, then the column is not converted.

The converted timestamps do not include time zone information. The meaning of the timestamp is based on the agent producing the data. Some agents output timestamps in local time and others in Greenwich mean time (GMT).

The CSV file name is also modified to use ISO timestamps. For example, the file name for table NTMEMORY with data from 2008/09/27 12:00 to 2008/09/27 19:00 would change from:

  • File name with proprietary timestamp: NTMEMORY_1080927120000000_1080927190000000_00000.csv

  • File name with ISO timestamp: NTMEMORY_2008-09-27T12-00-00-000_2008-09-27T19-00-00-000_00000.csv

The log file with error tracing indicates which columns containing proprietary timestamps were not converted due to invalid values. The log file with state tracing indicates which columns containing proprietary timestamps were converted to ISO timestamps.

KHD_CSV_ISO_DATE_FORMAT_EXCLUDE

Optional. Specify a list of column names, separated by a comma or space, to be excluded from the timestamp conversion to ISO format. The short table name must prefix the short column name, with a period separating the two names.

KHD_CSV_OUTPUT

Required. The directory where the CSV files are written. If the directory is not specified or does not exist, a message is logged to the trace file, an audit message is generated, and the CSV output is disabled.

KHD_CSV_OUTPUT_ACTIVATE

Required. Use Y to enable CSV output and N to disable CSV output. The default is N.

KHD_CSV_OUTPUT_LIST

Optional. The comma or space-separated list of table names (either the short name or the object name found in the ODI or "Disk capacity planning for historical data" table in the agent user’s guide) that is written to the CSV files, for example, KHD_CSV_OUTPUT_LIST=NTMEMORY,WTLOGCLDSK,NT_System.

If not supplied, all tables are written to CSV files. This setting is ignored for data tagged for analytical usage. The default is empty and has no maximum length.

This variable is ignored when the KHD_CSV_OUTPUT_TAGGED_ONLY variable is enabled.

KHD_CSV_OUTPUT_TAGGED_ONLY

Optional. If specified, only data that has been tagged for analytic use is written to the CSV files. The KHD_CSV_OUTPUT_LIST variable is ignored if this variable is enabled. The default is Y.

Use this variable in conjunction with the private situation USE= parameter in the <HISTORY> element . See Private situation XML specification.

Note: The data tagged for analytical use is not inserted into the Tivoli Data Warehouse.

If set to N, then enterprise data and private history (tagged and non-tagged) data are written to the CSV files.

KHD_CSV_MAXSIZE

Optional. The maximum total size in MB of the CSV files.

When this threshold is exceeded, the Warehouse Proxy agent stops writing data to the CSV files at the next evaluation interval. In IBM Tivoli Monitoring V6.3 and later, a warning message is logged in the Warehouse Proxy agent logs and an audit message is displayed. Files must be deleted to resume writing.

This checking cannot be disabled. The default is 10 MB.

The Warehouse Proxy agent workspaces are not updated to reflect these variables.

Monitor agent and monitoring server environment variables

Use the following environment variables to configure monitoring agents and monitoring servers that you want to export historical data to the Warehouse Proxy agent:

KHD_PURE_EVENT_UPLOAD

Optional. The interval in minutes to wait before historical data from pure events is uploaded to the Warehouse Proxy agent. This interval overrides the default setting of 30 minutes. The minimum value is 1 minute. There is no maximum value. If the historical data is stored at the monitoring agent, specify this environment variable in the monitoring agent's environment file. If the historical data is stored at the monitoring server, specify this environment variable in the monitoring servers's environment file. Default: "30".

If you specify a low interval, such as 1 minute, best practice is set KHD_HISTRETENTION=0 to minimize CPU and disk usage.

The Managed System Status > History Exports workspace is updated to reflect the Export Interval Time.

KHD_HISTRETENTION

Optional. Specifies the default retention period in hours for the short-term history files. This value can be used to reduce the amount of data that is kept on disk after a successful upload to the warehouse is performed. Default: "24".

If you configured private history, the value specified for the RETAIN= parameter in the <HISTORY> element overrides the KHD_HISTRETENTION value. If you do not specify a value for the RETAIN= parameter, then the KHD_HISTRETENTION value is used.

Specify KHD_HISTRETENTION=0 to delete the short term binary file after the export is complete. Deleting the short term binary file prevents space concerns and also speeds up the process of reading the files. After the short term binary file is deleted, you will not see short-term data in the Tivoli Enterprise Portal. If you set KHD_HISTRETENTION=0, then you must also change the variable KFW_REPORT_TERM_BREAK_POINT in the portal server environment file to ensure that historical queries in the portal server work correctly.


Parent topic:

Use your historical data for analytic usage

+

Search Tips   |   Advanced Search