$('a[name]').remove(); $('#ic-homepage__footer').before('

'); $("#tabs").tabs({ selected: 1 }); $("#ic-homepage__ic-tips").append( quickTipHTML() ); unhideOneProductTip(); $("#ic-homepage__product-tips").wrapInner('

'); $("#ic-homepage__feed-tips").wrapInner('

'); });

IBM Tivoli Monitoring > Version 6.3 > User's Guides > Agent Builder User's Guide IBM Tivoli Monitoring, Version 6.3

Monitor a log file

Configure your agent to receive data from a log file data source.

The agent monitors log files that are in the same locale and code page that the agent runs in.


Procedure

  1. On the Agent Initial Data Source page (Figure 1) or the Data Source Location page, click Logged Data in the Monitoring Data Categories area.

    Figure 1. Adding a log file

  2. In the Data Sources area, click A Log File.

  3. Click Next.

  4. On the Log File Information page (Figure 2), type the name of the log file you want to monitor in the Log File Information area. The file name must be fully qualified.

    Figure 2. Log File Information page

    1. Optional: Part of the log file name can come from a runtime configuration property. To create a log file name, click Insert Configuration Property and select a configuration property (Figure 1).

    2. Optional: The file can also be a dynamic file name. See (Dynamic file name support).

  5. In the Field Identification area, click one of the following options:

    Fixed number of characters

    When selected, limits the number of characters.

    With this option, each attribute is assigned the maximum number of characters it can hold from the log file. For example, if there are three attributes A, B, and C (in that order), and each attribute is a String of maximum length 20. Then, the first 20 bytes of the log record go into A, the second 20 into B, and the next 20 into C.

    Tab separator

    When selected, you can use tab separators.

    Space separator

    When selected, multiple concurrent spaces can be used as a single separator.

    Separator Text

    When selected, type in separator text.

    Begin and End Text

    When selected, type in both Begin and End text.

    XML in element

    When selected, type the name of the XML element to use as the record, or click Browse to define the element.

    If you clicked Browse, the XML Browser window is displayed (Figure 3). If you use the browse function, the Agent Builder identifies all possible attributes of the record by looking at the child tags and their attributes.

    Figure 3. XML Browser window

    Unless you click Advanced and fill out the information in that window, the following assumptions are made about information that you complete:

    • Only a log file at a time is monitored.

    • Each line of the log file contains all the fields necessary to fill the attributes to be defined.

    For more information about log file parsing and separators, see (Log file parsing and separators).

  6. Optional: Click Advanced on the Log File Information page to do the following using the Advanced Data Source Properties page (Figure 4):

    • Monitor more than one file, or monitor files with different names on different operating systems or monitor files with names that match regular expressions.

    • Draw a set of fields from more than one line in the log file.

    • Choose Event Filtering and Summarization Options.

    • Produce output summary information. This summary produces an additional attribute group at each interval. For more information about this attribute group, see (Log File Summary). This function is deprecated by the options available in the Event Information tab.

    Figure 4. Advanced Data Source Properties page, File Information

    1. To monitor more than 1-log file, click Add and type the name. If more than one file is listed, a unique label must be entered for each file. The label can be displayed as an attribute to indicate which file generated the record. It must not contain spaces.

    2. Optional: To select the operating systems on which each log file is to be monitored, follow these steps:

      1. Click in the Operating systems column for the log file.

      2. Click Edit.

      3. In the Operating Systems window, select the operating systems.

      4. Click OK to save your changes and return to the Advanced Data Source Properties page.

    3. Optional: Select File names match regular expression if the file name you are providing is a regular expression that is used to find the file instead of being a file name. See (ICU regular expressions). If you do not check this box, the name must be an actual file name. Alternatively it must be a pattern that follows the rules for file name patterns that are described in (Dynamic file name syntax).

    4. Optional: Select One directory element matches regular expression to match one subdirectory of the file name path with a regular expression. You can select this option only if you also selected File names match regular expression in the previous step.

      If regular expression meta characters are used in the path name, the meta characters can be used in only one subdirectory of the path. For example, you can specify /var/log/[0-9\.]*/mylog.* to have meta characters in one subdirectory. The [0-9\.]* results in matching any subdirectory of /var/log that consists solely of numbers and dots (.). The mylog.* results in matching any file names in those/var/log subdirectories that begin with mylog and are followed by zero or more characters.

      Because some operating systems use the backslash (\) as a directory separator it can be confused with a regular expression escape meta character. Because of this confusion forward slashes must always be used to indicate directories. For example, Windows files that are specified as C:\temp\mylog.* might mean the \t is a shorthand tab character. Therefore, always use forward slashes (/) on all operating systems for directory separators. The C:/temp/mylog.* example represents all files in the C:/temp directory that start with mylog.

    5. In the When multiple files match list, select one of the following options:

      • The file with the highest numerical value in the file name

      • The biggest file

      • The most recently-updated file

      • The most recently-created file

      • All files that match

      When you select All files that match, the agent identifies all files in the directory that match the dynamic file name pattern. The agent monitors updates to all of the files in parallel. Data from all files is intermingled during the data collection process. Its best to add an attribute by selecting Log file name in Record Field Information to correlate log messages to the log files that contain the log messages. Ensure that all files that match the dynamic file name pattern can be split into attributes in a consistent manner. If the log files selected cannot be coherently parsed, then its best to select Entire record in Record Field Information to define a single attribute. For more information about specifying Record Field Information for attributes, see step (8).

    6. Choose how the file is processed. With Process all records when the file is sampled, you can process all records in the entire file every time the defined sampling interval for the log monitor expires. The default interval is 60 seconds. This interval can be modified using the KUMP_DP_COPY_MODE_SAMPLE_INTERVAL environment variable (specifying a value in seconds). The same records are reported every time unless they are removed from the file. With this selection, event data is not produced when new records are written to the file. With Process new records appended to the file, you can process new records that are appended to the file while the agent is running. An event record is produced for every record added to the file. If the file is replaced (first record changes in any way), the file is processed and an event is produced for each record in the file.

      If appending records to an XML log file, the append records must contain a complete set of elements that are defined within the XML element you selected as Field Identification.

    7. If you chose to process new records that are appended to the file, you can also choose how new records are detected. With Detect new records when record count increases, new records can be detected when the number of records in the file increases, whether the size of the file changes. This feature is useful when an entire log file is pre-allocated before any records are written to the file. This option can be selected for files that are not pre-allocated, but it is less efficient than monitoring the size of the file. With Detect new records when the file size increases, you can determine when a new entry is appended to a file in the typical way. There might be a brief delay in recognizing that a monitored file is replaced.

    8. If you selected Detect new records when the file size increases, you can also choose how to process a file that exists when the monitoring agent starts. Ignore existing records disables event production for any record in the file at the time agent starts. Process ___ existing records from the file specifies production of an event for a fixed number of records from the end of the file at the time the agent starts. Process records not previously processed by the agent: Specifies for restart data to be maintained by the monitoring agent so the agent knows which records were processed the last time that it ran. Events are produced for any records that are appended to the file since the last time the agent was running. This option involves a little extra processing each time a record is added to the file.

    9. If you selected Process records not previously processed by the agent, you can choose what to do when the agent starts and apparently the existing file was replaced. Process all records if the file has been replaced: If information about the monitored file and the restart data information do not match, events are produced for all records in the file. Examples of mismatches include: The file name is different, the file creation-time is different, the file-size decreased, the file last modification time is earlier than before. Do not process records if the file has been replaced: If the information about the monitored file and the restart data information do not match, disables processing of existing records in the file.

    10. Click the Record Identification tab (Figure 5) to interpret multiple lines in the log file as a single logical record.

      If you select XML in element as the field identification on the Log File Information page, the Record Identification tab does not display.

      • Single line interprets each line as a single logical record.

      • Separator line you can enter a sequence of characters that identifies a line that separates one record from another.

        The separator line is not part of the previous or the next record.

      • Rule identifies a maximum number of lines that make up a record and optionally a sequence of characters that indicate the beginning or end of a record. With Rule, you can specify the following properties:

        • Maximum non-blank line defines the maximum number of non-blank lines that can be processed by a rule.

        • Type of rule: Can be one of:

          • No text comparison (The Maximum lines per record indicates a single logical record).

          • Identify the beginning of record (Marks the start of the single logical record).

          • Identify the end of record (Marks the end of the single logical record).

        • Offset: Specifies the location within a line where the Comparison String must occur.

        • Comparison Test: Can either be Equals, requiring a character sequence match at the specific offset, or Does not equal, indicating a particular character sequence does not occur at the specific offset.

        • Comparison String defines the character sequence to be compared.

      • Regular Expression identify a pattern that is used to indicate the beginning or end of a record. By using Regular Expression, you can specify the following properties:

        • Comparison String defines the character sequence to be matched.

          OR

        • Beginning or end of record:

          • Identify the beginning of record marks the start of the single logical record.

          • Identify the end of record marks the end of the single logical record.

      Figure 5. Advanced Data Source Properties page, Record Identification

    11. If you did select Process all records when the file is sampled earlier, click the Filter Expression tab. By clicking Filter Expression you can filter the data that is returned as rows based on the values of one or more attributes, configuration variables or both. If you selected Process new records appended to the file earlier you cannot create a filter expression. For more information about filtering data from an attribute group, see (Filtering attribute groups).

      Figure 6. Advanced Data Source Properties page, Filter Expression tab

    12. If you selected Process new records appended to the file earlier, click the Event Information tab Figure 7 to select Event Filtering and Summarization Options. See (Event filtering and summarization).

      The Summary tab can be present if the agent was created with an earlier version of Agent Builder. The summary tab is now deprecated by the Event Information tab

      Figure 7. The Event Information tab of the Advanced Data Source Properties page,

  7. Optional: Click Test Log File Settings on the Log File Information pageto start and test the agent (Figure 2). Click Test Log File Settings after you select the options for the log source. For more information about testing, see (Test log file attribute groups).

  8. Use the following steps if you did not use the test function earlier and you typed the log file name in the Log File Information area of the Log File Information page:

    1. Click Next to display the Attribute Information page and define the first attribute in the attribute group.

    2. Specify the information, on the Attribute Information page, and click Finish.

    When a log file attribute group is added to an agent at the default minimum Tivoli Monitoring version (6.2.1) or later, a Log File Status attribute group is included. For more information about the Log File Status attribute group, see (Log File Status attribute group).

    Figure 8. Attribute Information page

    Along with the fields applicable to all data sources, the Attribute Information page for the log file data source has some additional fields in the Record Field Information area.

    The Record Field Information fields are:

    Next field

    Shows the next field after parsing, using the delimiters from the attribute group (or special delimiters for this attribute from the Advanced dialog).

    Remainder of record

    Shows the rest of the record after previous attributes are parsed. This attribute is the last attribute, except for possibly the log file name or log file label.

    Entire record

    Shows the entire record, which can be the only attribute, except for possibly the log file name or log file label.

    Log file name

    Shows the name of the log file.

    Log file label

    Shows the label that is assigned to the file on the advanced panel.

    Use the Derived Attribute Details tab only if you want a derived attribute, and not an attribute directly from the log file.

  9. Click Advanced in the Record Field Information area to display the Advanced Log File Attribute Information page (Figure 9).

    Figure 9. Advanced Log File Attribute Information page

    1. In the Attribute Filters section, specify the criteria for data to be included or excluded. Filtering attributes can enhance the performance of your solution by reducing the amount of data processed. Click one or more of the attribute filters:

      • Inclusive indicates that the attribute filter set is an acceptance filter, meaning that if the filter succeeds, the record passes the filter, and is output.

      • Exclusive indicates that the attribute filter set is a rejection filter, meaning that if the attribute filter succeeds, the record is rejected, and is not output.

      • Match all filters indicates that all filters defined to the filter must match the attribute record in order for the filter to succeed.

      • Match any Filter indicates that if any of the filters that are defined to the filter match the attribute record, the filter succeeds.

    2. Use Add, Edit, and Remove to define the individual filters for an attribute filter set.

      Figure 10. Add Filter window

    3. To add a filter, follow these steps:

      1. Click Add, and complete the options in the Add Filter window (Figure 10) as follows:

        1. The Filter criteria section defines the base characteristics of the filter, including the following properties:

          • Start offset defines the position in the attribute string where the comparison is to begin.

          • Comparison string defines the pattern string against which the attribute is defined.

            Type a string, pattern, or regular expression that is used by the agent to filter the data read from the file. The records that match the filter pattern are eliminated from the records that are returned to the Tivoli Enterprise Portal, or are the only records returned. The result depends on whether you choose for the filter to be inclusive or exclusive.

          • Match entire value checks for an exact occurrence of the comparison string in the attribute string. Checking starts from the starting offset position.

          • Match any part of value checks for the comparison string anywhere in the attribute string. Checking starts from the starting offset position.

        2. The comparison string is a regular expression indicates that the comparison string is a regular expression pattern that can be applied against the attribute string.

          Regular expression-filtering support is provided using the International Components for Unicode (ICU) libraries to check whether the attribute value examined matches the specified pattern.

          To effectively use regular expression support, you must be familiar with the specifics of how ICU implements regular-expressions. This implementation is not identical to how regular expression support is implemented in Perl, grep, sed, Java regular expressions, and other implementations. See ICU regular expressions for guidance on creating regular expression filters.

        3. Define an override filter indicates that you want to provide a more specific filter comparison that overrides the base characteristics previously defined. This additional comparison string is used to reverse the filter result. When the filter is Inclusive, the override acts as an exclusion qualifier for the filter expression. When the filter is Exclusive, the override acts as an inclusion qualifier for the filter expression. (For more about Inclusive and Exclusive, see step 9, and the examples that follow). The override filter has the following properties:

          • Start offset defines the position in the attribute string where the comparison is to begin.

          • Comparison string defines the pattern string against which the attribute is matched.

            Type a regular expression that is used by the agent to filter the data read from the file. The records that match the filter pattern are eliminated from the records that are returned to the Tivoli Enterprise Portal, or are the only records returned. The result depends on whether you choose for the filter to be inclusive or exclusive.

        4. Replacement value can be used to alter the raw attribute string with a new value. See ICU regular expressions for more details about special characters that can be used.

        5. Replace first occurrence replaces the first occurrence that is matched by the comparison string with new text.

        6. Replace all occurrences replaces all occurrences that are matched by the comparison string with new text.

      2. Click OK.

      Figure 11. Add Filter example 1

      If the attribute string is abc is easy as 123, then the replaced string that is displayed in the Tivoli Enterprise Portal as 123 is not as easy as abc.

      Figure 12. Add Filter example 2

      If the attribute string is Unrecoverable Error reading from disk, and the filter is Inclusive, then the attribute is displayed in the Tivoli Enterprise Portal. If the attribute string is No Errors Found during weekly backup and the filter is Inclusive, then the attribute is not displayed in the Tivoli Enterprise Portal.

    4. In the Field Identification section of the Advanced Log File Attribute Information page (Figure 9), specify how to override the attribute group field delimiters for this one attribute only. Click one of the attribute filters, and complete the required fields for the option:

      • Number of characters: Enter the limit for the number of characters.

      • Tab separator specifies the use of tab separators.

      • Separator Text: Enter the separator text that you want to use.

      • Begin and End Text Enter both Begin text and End text.

    5. In the Summary section of the Advanced Log File Attribute Information page (Figure 9), click the Include attribute in summary attribute group check box to add the attribute to the summary attribute group. This attribute group is produced when a user turns on log attribute summarization.

    6. Click OK.

  10. If you used the test function in step (7), the Select key attributes page is displayed. On the Select key attributes page, select key attributes or indicate that this data source produces only one data row. See (Selecting key attributes).

  11. Do one of the following steps:

    • If you are using the New Agent wizard, click Next.

    • Click Finish to save the data source and open the Agent Editor.

    When a log file attribute group is added to an agent with the default minimum Tivoli Monitoring version (6.2.1) or later, a Log File Status attribute group is included. For more information about the Log File Status attribute group, see (Log File Status attribute group).


+

Search Tips   |   Advanced Search