Agent troubleshooting

$('a[name]').remove(); $('#ic-homepage__footer').before('

'); $("#tabs").tabs({ selected: 1 }); $("#ic-homepage__ic-tips").append( quickTipHTML() ); unhideOneProductTip(); $("#ic-homepage__product-tips").wrapInner('

'); $("#ic-homepage__feed-tips").wrapInner('

'); });

IBM Tivoli Monitoring > Version 6.3 > User's Guides > Log File Agent User's Guide > Troubleshooting > Problems and workarounds IBM Tivoli Monitoring, Version 6.3

Agent troubleshooting

A problem can occur with the agent after it has been installed.
Table 1 contains problems and solutions that can occur with the agent after it is installed.

Agent problems and solutions

Problem Solution
Log data accumulates too rapidly. Check the RAS trace option settings, which are described in Set RAS trace parameters using the GUI. The trace option settings that you can set on the KBB_RAS1= and KDC_DEBUG= lines potentially generate large amounts of data.
When using the itmcmd agent commands to start or stop this monitoring agent, you receive the following error message:
MKCIIN0201E Specified product is not configured.
Include the command option -o to specify the instance to start or stop. The instance name must match the name used for configuring the agent. For example:
./itmcmd agent -o Test1 start lo
For more information about using the itmcmd commands, see the Command Reference.
A configured and running instance of the monitoring agent is not displayed in the Tivoli Enterprise Portal, but other instances of the monitoring agent on the same system are displayed in the portal. IBM Tivoli Monitoring products use Remote Procedure Call (RPC) to define and control product behavior. RPC is the mechanism that a client process uses to make a subroutine call (such as GetTimeOfDay or ShutdownServer) to a server process somewhere in the network. Tivoli processes can be configured to use TCP/UDP, TCP/IP, SNA, and SSL as the protocol (or delivery mechanism) for RPCs that you want.
IP.PIPE is the name given to Tivoli TCP/IP protocol for RPCs. The RPCs are socket-based operations that use TCP/IP ports to form socket addresses. IP.PIPE implements virtual sockets and multiplexes all virtual socket traffic across a single physical TCP/IP port (visible from the netstat command).
A Tivoli process derives the physical port for IP.PIPE communications based on the configured, well-known port for the hub Tivoli Enterprise Monitoring Server. (This well-known port or BASE_PORT is configured using the 'PORT:' keyword on the KDC_FAMILIES / KDE_TRANSPORT environment variable and defaults to '1918'.)
The physical port allocation method is defined as (BASE_PORT + 4096*N), where N=0 for a Tivoli Enterprise Monitoring Server process and N={1, 2, ..., 15} for another type of monitoring server process. Two architectural limits result as a consequence of the physical port allocation method:

No more than one Tivoli Enterprise Monitoring Server reporting to a specific Tivoli Enterprise Monitoring Server hub can be active on a system image.

No more than 15 IP.PIPE processes can be active on a single system image.

A single system image can support any number of Tivoli Enterprise Monitoring Server processes (address spaces) if each Tivoli Enterprise Monitoring Server on that image reports to a different hub. By definition, one Tivoli Enterprise Monitoring Server hub is available per monitoring enterprise, so this architecture limit has been reduced to one Tivoli Enterprise Monitoring Server per system image.
No more than 15 IP.PIPE processes or address spaces can be active on a single system image. With the first limit expressed earlier, this second limitation refers specifically to Tivoli Enterprise Monitoring Agent processes: no more than 15 agents per system image.
Continued on next row.
Continued from previous row.
This limitation can be circumvented (at current maintenance levels, IBM Tivoli Monitoring V6.1, Fix Pack 4 and later) if the Tivoli Enterprise Monitoring Agent process is configured to use the EPHEMERAL IP.PIPE process. (This process is IP.PIPE configured with the 'EPHEMERAL:Y' keyword in the KDC_FAMILIES / KDE_TRANSPORT environment variable). The number of ephemeral IP.PIPE connections per system image has no limitation. If ephemeral endpoints are used, the Warehouse Proxy agent is accessible from the Tivoli Enterprise Monitoring Server associated with the agents using ephemeral connections either by running the Warehouse Proxy agent on the same computer or using the Firewall Gateway feature. (The Firewall Gateway feature relays the Warehouse Proxy agent connection from the Tivoli Enterprise Monitoring Server computer to the Warehouse Proxy agent computer if the Warehouse Proxy agent cannot coexist on the same computer.)
I cannot find my queries. Agents that include subnodes display their queries within the element in the Query Editor list that represents the location of the attribute group. The queries are most often found under the name of the subnode, not the name of the agent.
No events are visible in the Tivoli Enterprise Portal even though the agent is started. This problem can be caused be one of these issues:

The agent is not configured to send events to IBM Tivoli Monitoring:
Ensure that the Send ITM Events configuration value is set to "yes" in the command or checked in the GUI.

The configuration file cannot be found. The performance object status attribute group reports CONF FILE DOES NOT EXIST:
Ensure the value for the configuration file configuration setting is correct. It must be a fully qualified path to the configuration file and the agent must have sufficient access rights to read the file.

The format file cannot be found. The performance object status attribute group reports CANNOT OPEN FORMAT FILE:
Ensure the value for the format file configuration setting is correct. It must be a fully qualified path to the format file and the agent must have sufficient access rights to read the file.

There is a parsing error in the format file. The performance object status attribute group reports FORMAT FILE SYNTAX ERROR. The agent's error log identifies the line of the format file where the parsing error occurred:
Correct the syntax error in the format file.

No patterns in the format file match records read from the log source:
Configure the agent instance to use an unmatch file and examine the contents of the unmatch log. Use a regular expression test tool to help write expressions that match the records written to the unmatch log. Update the format file.

The KLO_Log_Agent_Config_Error situation is true for the Tivoli Log File Agent instance. The KLO_Log_Agent_Config_Error situation is true if any of the following issues occur:

The agent is not configured to send events to IBM Tivoli Monitoring:
Ensure that the Send ITM Events configuration value is set to "yes" in the command or checked in the GUI.

The configuration file cannot be found. The performance object status attribute group reports CONF FILE DOES NOT EXIST:
Ensure the value for the configuration file configuration setting is correct. It must be a fully qualified path to the configuration file and the agent must have sufficient access rights to read the file.

The format file cannot be found. The performance object status attribute group reports CANNOT OPEN FORMAT FILE:
Ensure the value for the format file configuration setting is correct. It must be a fully qualified path to the format file and the agent must have sufficient access rights to read the file.

There is a parsing error in the format file. The performance object status attribute group reports FORMAT FILE SYNTAX ERROR. The agent's error log identifies the line of the format file where the parsing error occurred:
Correct the syntax error in the format file.

No patterns in the format file match records read from the log source:
Configure the agent instance to use an unmatch file and examine the contents of the unmatch log. Use a regular expression test tool to help write expressions that match the records written to the unmatch log. Update the format file.

Some events are initially visible in the Tivoli Enterprise Portal, but no new events are generated even though the log sources contain records that match expressions in the format file. If the agent is configured to send EIF events, but no EIF receiver is configured or running (such as an OMNIbus probe) to receive the events, the agent's EIF reception log fills and no events can be processed.
If you do not want EIF events sent, complete the following steps:

Stop the agent.

Reconfigure the agent and set the Send EIF Events to OMNIbus configuration value to "No".

Start the agent.

If you want EIF events sent, complete the following steps:

Ensure that the EIF receiver is running.

Verify the EIF settings in the agent's configuration file.

If too many events are sent, the EIF event cache may not be large enough to handle the event storm. Increase the value of the BufEvtMaxSize configuration value in the agent's configuration file.

syslogd does not support named pipes on some HP-UX operating systems
The Automatically initialize UNIX syslog configuration option uses syslogd. Because some HP-UX operating systems do not support named pipes, the Automatically initialize UNIX syslog feature does not work on these HP-UX operating systems.
There is no workaround.
An agent configured to send EIF events, stops processing events at a time when there are a large number of events generated and appears to hang. The EIF event cache may not be large enough to handle the event storm. Increase the value of the BufEvtMaxSize configuration value in the agent's configuration file.
Log record does not match the expected regular expression. The event generated does not have the expected event class or event attributes. Expressions are processed from the end of the format file to the beginning. The first expression that matches a record is the one that is used. Ensure that the more generic expressions are at the beginning of the file and the more specific ones at the end.
While using LogSources or RegexLogSources, the file is not being monitored. Set the KBB_RAS1=ERROR (UNIT:kumpdcm STATE) creates a trace entry every time a file is matched against the LogSources pattern and FileComparisonMethod. Example output: Directory /logs/data/ File notsmoke.log did not match pattern smoke.*\.log File <smoke03.log> skipped, not more recently updated than previous file <smoke01.log>
When using RegexLogSources, if more than one subdirectory has meta characters how can you tell what regular expression and filename is being used. Log File Status attribute group shows information about each log file being monitored. For each file being monitored, the regular expression pattern used to select the file is shown. If the log file pattern with the RegexLogSources was specified incorrectly, the regular expression pattern shown in the Log File Status attribute group is different from the one specified in the configuration (.conf) file and a warning trace entry is be written.
On Windows systems, when the LogSources and RegexLogSources statement in the .conf file specify files in the root directory without the drive letter, the files to be monitored are not found. For example:
LogSources=\temp*
RegexLogSources=/.*/reg4ex/.*.log
Include the drive letter of the file path in the LogSources and RegexLogSources statement. For example: LogSources=C:\temp* and RegexLogSources=C:/.*/reg4ex/.*.log. If a drive letter is not specified for the root directory of LogSources or RegexLogSources, the agent assumes that the drive letter is C: even if the system is configured to have a different default drive letter.
If the EIF configuration is set to connection oriented, ConnectionMode=co or ConnectionMode=connection_oriented in the Log File agent configuration file, and the EIF receiver is down when the agent starts, the remote receiver does not respond for approximately two minutes. The agent is not monitoring the logs during this time. If Profiles/subnodes are in use, and each .conf file is configured the same way, these delays are sequential. To avoid this issue, ensure the EIF receiver is up and running when the agent starts, or use the connection_less option for event delivery.
On HP/UX 11.11, some systems default to only allowing 65 threads per process. The Log File Agent uses a thread for each file that it is monitoring, as well as a number of other threads performing background housekeeping tasks. When using subnodes or monitoring a large number of files, it is possible to exceed this operating system-imposed limit, which causes the monitoring of some files to fail. To avoid this issue, use the HP/UX sam tool to configure the kernel parameter max_thread_proc to a larger value. The recommended value is 500 to allow enough threads for a large configuration.
Not all log files that are specified are being monitored.
Check the Log File Status attribute group. It shows information about each log file that is being monitored. In particular:

Verify that the file is one of the log files that is being monitored. If it is being monitored, verify that the File Status attribute is OK (0). If it is OK, verify that the Current File Position is less than the Current File Size as the log file agent might have already read the entire file.

Verify that for one of the files that is being monitored, the RegEx Pattern attribute matches the file specified in the configuration (.conf) file that represents the missing file.

If all log files are not in the Log File Status attribute group, check the trace log files for a limit on system resources. For example, each log file that is being monitored is a separate thread and a UNIX system might have a ulimit -u (max user processes) which is too small.

**** File server process startup failed for file address file name

**** MultiThreaded File server process startup failed for file file name

**** File server process startup failed for file name

There may be issues with Czech language support when viewing help files on some Firefox browser versions. Use Firefox browser version 18 or later.

Parent topic:
Problems and workarounds

+
Search Tips | Advanced Search

Problem	Solution
Log data accumulates too rapidly.	Check the RAS trace option settings, which are described in Set RAS trace parameters using the GUI. The trace option settings that you can set on the KBB_RAS1= and KDC_DEBUG= lines potentially generate large amounts of data.
When using the itmcmd agent commands to start or stop this monitoring agent, you receive the following error message: MKCIIN0201E Specified product is not configured.	Include the command option -o to specify the instance to start or stop. The instance name must match the name used for configuring the agent. For example: ./itmcmd agent -o Test1 start lo For more information about using the itmcmd commands, see the Command Reference.
A configured and running instance of the monitoring agent is not displayed in the Tivoli Enterprise Portal, but other instances of the monitoring agent on the same system are displayed in the portal.	IBM Tivoli Monitoring products use Remote Procedure Call (RPC) to define and control product behavior. RPC is the mechanism that a client process uses to make a subroutine call (such as GetTimeOfDay or ShutdownServer) to a server process somewhere in the network. Tivoli processes can be configured to use TCP/UDP, TCP/IP, SNA, and SSL as the protocol (or delivery mechanism) for RPCs that you want. IP.PIPE is the name given to Tivoli TCP/IP protocol for RPCs. The RPCs are socket-based operations that use TCP/IP ports to form socket addresses. IP.PIPE implements virtual sockets and multiplexes all virtual socket traffic across a single physical TCP/IP port (visible from the netstat command). A Tivoli process derives the physical port for IP.PIPE communications based on the configured, well-known port for the hub Tivoli Enterprise Monitoring Server. (This well-known port or BASE_PORT is configured using the 'PORT:' keyword on the KDC_FAMILIES / KDE_TRANSPORT environment variable and defaults to '1918'.) The physical port allocation method is defined as (BASE_PORT + 4096*N), where N=0 for a Tivoli Enterprise Monitoring Server process and N={1, 2, ..., 15} for another type of monitoring server process. Two architectural limits result as a consequence of the physical port allocation method: No more than one Tivoli Enterprise Monitoring Server reporting to a specific Tivoli Enterprise Monitoring Server hub can be active on a system image. No more than 15 IP.PIPE processes can be active on a single system image. A single system image can support any number of Tivoli Enterprise Monitoring Server processes (address spaces) if each Tivoli Enterprise Monitoring Server on that image reports to a different hub. By definition, one Tivoli Enterprise Monitoring Server hub is available per monitoring enterprise, so this architecture limit has been reduced to one Tivoli Enterprise Monitoring Server per system image. No more than 15 IP.PIPE processes or address spaces can be active on a single system image. With the first limit expressed earlier, this second limitation refers specifically to Tivoli Enterprise Monitoring Agent processes: no more than 15 agents per system image. Continued on next row.
Continued from previous row.	This limitation can be circumvented (at current maintenance levels, IBM Tivoli Monitoring V6.1, Fix Pack 4 and later) if the Tivoli Enterprise Monitoring Agent process is configured to use the EPHEMERAL IP.PIPE process. (This process is IP.PIPE configured with the 'EPHEMERAL:Y' keyword in the KDC_FAMILIES / KDE_TRANSPORT environment variable). The number of ephemeral IP.PIPE connections per system image has no limitation. If ephemeral endpoints are used, the Warehouse Proxy agent is accessible from the Tivoli Enterprise Monitoring Server associated with the agents using ephemeral connections either by running the Warehouse Proxy agent on the same computer or using the Firewall Gateway feature. (The Firewall Gateway feature relays the Warehouse Proxy agent connection from the Tivoli Enterprise Monitoring Server computer to the Warehouse Proxy agent computer if the Warehouse Proxy agent cannot coexist on the same computer.)
I cannot find my queries.	Agents that include subnodes display their queries within the element in the Query Editor list that represents the location of the attribute group. The queries are most often found under the name of the subnode, not the name of the agent.
No events are visible in the Tivoli Enterprise Portal even though the agent is started.	This problem can be caused be one of these issues: The agent is not configured to send events to IBM Tivoli Monitoring: Ensure that the Send ITM Events configuration value is set to "yes" in the command or checked in the GUI. The configuration file cannot be found. The performance object status attribute group reports CONF FILE DOES NOT EXIST: Ensure the value for the configuration file configuration setting is correct. It must be a fully qualified path to the configuration file and the agent must have sufficient access rights to read the file. The format file cannot be found. The performance object status attribute group reports CANNOT OPEN FORMAT FILE: Ensure the value for the format file configuration setting is correct. It must be a fully qualified path to the format file and the agent must have sufficient access rights to read the file. There is a parsing error in the format file. The performance object status attribute group reports FORMAT FILE SYNTAX ERROR. The agent's error log identifies the line of the format file where the parsing error occurred: Correct the syntax error in the format file. No patterns in the format file match records read from the log source: Configure the agent instance to use an unmatch file and examine the contents of the unmatch log. Use a regular expression test tool to help write expressions that match the records written to the unmatch log. Update the format file.
The KLO_Log_Agent_Config_Error situation is true for the Tivoli Log File Agent instance.	The KLO_Log_Agent_Config_Error situation is true if any of the following issues occur: The agent is not configured to send events to IBM Tivoli Monitoring: Ensure that the Send ITM Events configuration value is set to "yes" in the command or checked in the GUI. The configuration file cannot be found. The performance object status attribute group reports CONF FILE DOES NOT EXIST: Ensure the value for the configuration file configuration setting is correct. It must be a fully qualified path to the configuration file and the agent must have sufficient access rights to read the file. The format file cannot be found. The performance object status attribute group reports CANNOT OPEN FORMAT FILE: Ensure the value for the format file configuration setting is correct. It must be a fully qualified path to the format file and the agent must have sufficient access rights to read the file. There is a parsing error in the format file. The performance object status attribute group reports FORMAT FILE SYNTAX ERROR. The agent's error log identifies the line of the format file where the parsing error occurred: Correct the syntax error in the format file. No patterns in the format file match records read from the log source: Configure the agent instance to use an unmatch file and examine the contents of the unmatch log. Use a regular expression test tool to help write expressions that match the records written to the unmatch log. Update the format file.
Some events are initially visible in the Tivoli Enterprise Portal, but no new events are generated even though the log sources contain records that match expressions in the format file.	If the agent is configured to send EIF events, but no EIF receiver is configured or running (such as an OMNIbus probe) to receive the events, the agent's EIF reception log fills and no events can be processed. If you do not want EIF events sent, complete the following steps: Stop the agent. Reconfigure the agent and set the Send EIF Events to OMNIbus configuration value to "No". Start the agent. If you want EIF events sent, complete the following steps: Ensure that the EIF receiver is running. Verify the EIF settings in the agent's configuration file. If too many events are sent, the EIF event cache may not be large enough to handle the event storm. Increase the value of the BufEvtMaxSize configuration value in the agent's configuration file.
syslogd does not support named pipes on some HP-UX operating systems The Automatically initialize UNIX syslog configuration option uses syslogd. Because some HP-UX operating systems do not support named pipes, the Automatically initialize UNIX syslog feature does not work on these HP-UX operating systems.	There is no workaround.
An agent configured to send EIF events, stops processing events at a time when there are a large number of events generated and appears to hang.	The EIF event cache may not be large enough to handle the event storm. Increase the value of the BufEvtMaxSize configuration value in the agent's configuration file.
Log record does not match the expected regular expression. The event generated does not have the expected event class or event attributes.	Expressions are processed from the end of the format file to the beginning. The first expression that matches a record is the one that is used. Ensure that the more generic expressions are at the beginning of the file and the more specific ones at the end.
While using LogSources or RegexLogSources, the file is not being monitored.	Set the `KBB_RAS1=ERROR (UNIT:kumpdcm STATE)` creates a trace entry every time a file is matched against the LogSources pattern and FileComparisonMethod. Example output: Directory /logs/data/ File notsmoke.log did not match pattern smoke.*\.log File <smoke03.log> skipped, not more recently updated than previous file <smoke01.log>
When using RegexLogSources, if more than one subdirectory has meta characters how can you tell what regular expression and filename is being used.	Log File Status attribute group shows information about each log file being monitored. For each file being monitored, the regular expression pattern used to select the file is shown. If the log file pattern with the RegexLogSources was specified incorrectly, the regular expression pattern shown in the Log File Status attribute group is different from the one specified in the configuration (.conf) file and a warning trace entry is be written.
On Windows systems, when the LogSources and RegexLogSources statement in the .conf file specify files in the root directory without the drive letter, the files to be monitored are not found. For example: `LogSources=\temp` `RegexLogSources=/./reg4ex/.*.log`	Include the drive letter of the file path in the LogSources and RegexLogSources statement. For example: `LogSources=C:\temp` and `RegexLogSources=C:/./reg4ex/.*.log`. If a drive letter is not specified for the root directory of LogSources or RegexLogSources, the agent assumes that the drive letter is C: even if the system is configured to have a different default drive letter.
If the EIF configuration is set to connection oriented, `ConnectionMode=co` or `ConnectionMode=connection_oriented` in the Log File agent configuration file, and the EIF receiver is down when the agent starts, the remote receiver does not respond for approximately two minutes. The agent is not monitoring the logs during this time. If Profiles/subnodes are in use, and each .conf file is configured the same way, these delays are sequential.	To avoid this issue, ensure the EIF receiver is up and running when the agent starts, or use the `connection_less` option for event delivery.
On HP/UX 11.11, some systems default to only allowing 65 threads per process. The Log File Agent uses a thread for each file that it is monitoring, as well as a number of other threads performing background housekeeping tasks. When using subnodes or monitoring a large number of files, it is possible to exceed this operating system-imposed limit, which causes the monitoring of some files to fail.	To avoid this issue, use the HP/UX sam tool to configure the kernel parameter max_thread_proc to a larger value. The recommended value is 500 to allow enough threads for a large configuration.
Not all log files that are specified are being monitored.	Check the Log File Status attribute group. It shows information about each log file that is being monitored. In particular: Verify that the file is one of the log files that is being monitored. If it is being monitored, verify that the File Status attribute is OK (0). If it is OK, verify that the Current File Position is less than the Current File Size as the log file agent might have already read the entire file. Verify that for one of the files that is being monitored, the RegEx Pattern attribute matches the file specified in the configuration (.conf) file that represents the missing file. If all log files are not in the Log File Status attribute group, check the trace log files for a limit on system resources. For example, each log file that is being monitored is a separate thread and a UNIX system might have a ulimit -u (max user processes) which is too small. ** File server process startup failed for file address file name MultiThreaded File server process startup failed for file file name ** File server process startup failed for file name
There may be issues with Czech language support when viewing help files on some Firefox browser versions.	Use Firefox browser version 18 or later.