IBM Tivoli Monitoring > Version 6.3 Fix Pack 2 > Administrator's Guide > Manage historical data

IBM Tivoli Monitoring, Version 6.3 Fix Pack 2

Historical data collection configuration

After your configured historical data collections begin saving data samples, make provisions to manage it. Without additional action, the history data files can grow unchecked, using up valuable disk space.

Define historical data collection

The Historical Collection Configuration window is available through the Tivoli Enterprise Portal. You can specify the collection and storage of historical data at either the Tivoli Enterprise Monitoring Server or at the remote system where the monitoring agent is installed. For flexibility in using historical data collection, you can:

  • Configure multiple collections for the same attribute group. Each collection has a different distribution, can have a different collection interval, and can have a different warehouse interval.

  • Reduce the amount of data collected to only what passes a filter that you create. For example, collect only the data samples with processor busy time greater than 80%.

  • Configure a historical collection for all managed systems on a specific monitoring server when distribution is set to "Managing System (TEMS)", and for any managed system or set of managed systems when distribution is set to "Managed System (Agent)".

  • Set the Collection Location to save short-term history at the monitoring server or at the monitoring agent for each historical collection.

  • Set the Collection Interval, how often to send data samples to the short-term history file, from once a minute to once a day for each historical collection.

  • Set the Warehouse Interval, how often to save data into the Tivoli Data Warehouse, from every 15 minutes to once a day for each historical collection.

  • Determine how and when to summarize and prune the data that is stored in the data warehouse. Summarization and pruning is configured for each attribute group that has one or more historical collections defined.

  • Start collection on a managed system by adding it (or a managed system group it belongs to) to the distribution list of a historical collection or to a historical configuration group that the collection is a member of.

  • Stop collection on a managed system by removing it (or a managed system group it belongs to) from the distribution list of a historical collection or from a historical configuration group that the collection is a member of.

  • Create historical configuration groups with a distribution list and assign collections to the group that you want to use the distribution.

Define historical data collections from the command line

The historical data collection can also be configured using the command-line interface tacmd hist commands:

If you have a test environment, you can write scripts that use tacmds for configuring historical data collections and run the script on other test computers or on the production system so that you do not need to repeat the same configuration for each system. For more information about these commands, see IBM Tivoli Monitoring Command Reference.

Avoid redundant data collection

It is possible to collect data twice or more for the same attribute group on a managed system. This happens if you have configured multiple historical collections for the same attribute group and distribute them to the same managed system. Not only does this create more data traffic and use storage space unnecessarily, your summarization values are skewed. The skewing happens because the additional values sent by multiple collections for the same attribute group are factored into the summarization calculation.

For a given historical collection, you need not be concerned about inadvertently assigning the same managed system to the historical collection distribution multiple times. The historical function is aware of the managed systems the collection is distributed to and collects a data sample only once for each managed system. A managed system is included in the distribution of a historical data collection when it is:

  • directly referenced in the collection definition

  • in a managed system group that is referenced in the collection definition

  • in the distribution of a historical configuration group that the historical collection is a member of

  • in a managed system group that is in the distribution of a historical configuration group that the historical collection is a member of

Create filter formulas for granular data collection

The History Collection Configuration editor has a Filter tab with a formula editor for writing filter criteria that specifies the data to collect. Historical collection of a data sample only occurs if the values in the data row meet the filter criteria. For example, if the attribute value for % Disk Write Time is greater than 50%, the data sample is saved to short-term history; otherwise the sample is not saved.

The filter criteria is configurable for each collection definition. Applying filters to historical data collection can help reduce network traffic and wasted disk space and improve summarization and pruning performance.

Be aware that filtered data collection can affect the results of trending calculations that are performed with chart baselining and situation modeling (see Chart baselines and Modeling conditions for situations), and of query-based views that include historical data (see Set a time span to display). For example, a filter to collect only rows where the process uses 80% or more CPU means that a calculation of average values will be only of values 80% and higher, rather than of all values.

Trimming short-term history files

If you have chosen to upload data through the warehouse proxy to the Tivoli Data Warehouse, then the short-term history files on the monitoring server or monitoring agent are automatically trimmed after upload.

Manage System (TEMS) collection type

Historical data collection can be specified for individual monitoring servers, products, and attribute groups. However, all agents of the same type that report directly to the same monitoring server must have the same history collection options. Also, for a given attribute group, the same history collection options are applied to all monitoring servers for which that collection is currently enabled.

Collection location

The Collection Location is where the short-term historical data file resides: at the TEMA (Tivoli Enterprise Monitoring Agent) or the TEMS (Tivoli Enterprise Monitoring Server). The default location is TEMA, which minimizes the performance impact on the monitoring server from historical data management. However, there are some product and attribute group combinations that are only collected at a specific place, either the monitoring server or the monitoring agent.

On OMEGAMON XE products, the persistent data store is used to store short-term history, so it must be configured at the collection location. For any given agent, do not vary the collection location: collect all historical data for the product either at the monitoring agent or monitoring server. For agents that are configured in the same address space as the monitoring servers (required for OMEGAMON XE for z/OS and OMEGAMON XE for Storage on z/OS), configure the persistent data store in the same address space, and specify TEMS as the collection location.

Aggregate and prune warehouse data

The Summarization and Pruning Agent is a mechanism for managing data in the Tivoli Data Warehouse. The data in the warehouse is a historical record of activity and conditions in the enterprise. Summarization of the data is the process of aggregating your historical data into time-based categories, for example, hourly, daily, weekly, and so on. Summarizing data allows you to perform historical analysis of the data over time. Pruning of the data keeps the database to manageable size and thus improves performance. Pruning of the database should be performed at regular intervals.

You can run only one summarization and pruning agent even if you have multiple monitoring servers that are sharing a single Tivoli Data Warehouse database. Running multiple summarization and pruning agents causes database deadlocks and conflicts because the multiple instances might attempt to summarize or prune the data in the tables simultaneously.

Convert short-term history files to delimited flat files

If you choose not to use the Tivoli Data Warehouse, then you must institute roll-off jobs to regularly convert and empty out the history data files. Roll-off programs are provided. In addition to trimming the history data files, these scripts produce flat files which can be used with third-party tools to produce trend analysis reports and graphics. There is also an environment variable for setting the maximum size of history files.

See Limit the growth of short-term history files.

Some attribute groups are ineligible for historical data collection

Some agents do not enable collection of history data for all of their attribute groups. This is because the product development team for that agent has determined that collecting history data for certain attribute groups is not appropriate or might have a detrimental effect on performance. This might be because of the vast amount of data that is generated. Therefore, for each product, only attribute groups that are available for history collection are shown in the History Collection Configuration window when you click a Monitored Application.


Parent topic:

Manage historical data

+

Search Tips   |   Advanced Search