Using IBM Rational Performance Tester: Resource monitoring Part 3, Monitoring with UNIX/Linux rstatd
Monitor using rstatd for Linux/UNIX systems
The rstat daemon (rpc.rstatd) returns performance statistics for UNIX-based operating systems. The performance statistics are retrieved from the kernel itself.
The rstat daemon may already be installed and running on most Solaris and Linux installations, however, this configuration will depend on the vendor and distribution. The rstat daemon is normally started by the inetd daemon.
Performance counters
The statistics monitored are restricted to fifteen performance counters, which include processor, disk, memory, and network utilization.
| Performance counter | Description |
|---|---|
| Percentage User CPU Time | Percentage user time is the percentage of non-idle processor time spent in user mode. |
| Percentage IOWAIT CPU Time | Percentage IOWAIT time is the percentage of non-idle processor time spent waiting for I/O to complete. |
| Percentage of System CPU Time | Percentage system time is the percentage of non-idle processor time spent in system mode. |
| Percentage of Idle CPU Time | Percentage of time that the processor is idle. |
| Total Disk Transfers per second | Total number of disk transfers on each of the disk interfaces (per second). |
| Total VM Pages Paged IN per second | Total number of pages paged in per second. (Paging vs. swapping: see Note 1, below) |
| Total VM Pages Paged OUT per second | Total number of pages paged out per second. (Paging vs. swapping) |
| Total VM Pages Swapped IN per second | Total number of pages swapped in per second. (Paging vs. swapping) |
| Total VM Pages Swapped OUT per second | Total number of pages swapped out per second. (Paging vs. swapping) |
| Total Interrupts per second | Total number of interrupts per second. |
| Total Inbound packets on all interfaces per second | Total number of inbound packets on all interfaces per second |
| Total Inbound errors on all interfaces per second | Total number of inbound errors on all interfaces per second |
| Total Outbound packets on all interfaces per second | Total number of outbound packets on all interfaces per second |
| Total Outbound errors on all interfaces per second | Total number of outbound errors on all interfaces per second |
| Total collisions seen on all interfaces per second | Total number of collisions seen on all interfaces per second |
| Total context switches per second | Total number of context switches per second |
| Average number of jobs in run queue (1 minute average) | Number of jobs in the run queue averaged over the last 1 minute. See Note 2, below |
| Average number of jobs in run queue (5 minute average) | Number of jobs in the run queue averaged over the last 5 minutes. |
| Average number of jobs in run queue (15 minute average) | Number of jobs in the run queue averaged over the last 15 minutes. |
- Paging is a light-weight mechanism whereby the operating system reads/writes pages of memory from/to the disk; swapping is similar to paging, but is more heavy-weight in that entire processes can have all their pages read/written from/to disk.
- The jobs on the run queue are the processes that are ready to run on the processor.
Data collection
The rstat daemon is a native application that specifically operates on UNIX-based systems, such as Solaris, HP-UX, AIX, and Linux. Performance Tester integrates with the daemon through a Java-based RPC client, which is responsible for marshalling transactions to and processing the results from a rstat daemon. The RPC client is bundled within an Eclipse plug-in that extends the Resource Monitoring platform within Performance Tester.
When performance counters are selected to be monitored, a Remote Service Monitor (RSM) program polls the counters at an interval and sends alerts to the client upon detection. The RSM program sends resource data to the client at reporting intervals set by the client or system administrator.
Depending on the setup of your account and the UNIX-based operating system, the system administrator might control the location and maintenance of the RSM programs that monitor your account. The RSM program obtains the performance data from the rstatd server, which returns performance statistics obtained from the kernel.
The version of rstatd that is supported in Performance Tester is the RSTATVERS_TIME version, or version 3. The rstatd is backwards compatible, therefore, versions greater than 3 will support the RSTATVERS_TIME edition. Historically, this version has been in existence well before 1995.
System configuration
In this section we describe how to configure the system for the rstat daemon.
Installing rstatd on Linux
The rstat daemon is freely available on the Internet for the Linux platform, and is already bundled in most UNIX-based platforms. The following are installation instructions for rstatd on Linux using the open source version of rstatd available on SourceForge:
- Go to the Rstatd 4 Linux site and download the latest version:
- Extract the archive downloaded from the previous step, and follow the instructions in the install file. This file will be available after extracting the archive. Installation will require tools such as make and gcc, which are usually installed by default with most Linux distributions.
- The next step requires verifying that the rstat daemon gets started through inetd daemon (or xinetd on some Linux distributions). Using a terminal, change directories to /etc/xinetd.d and create a file called rstatd, with the following contents:
Creating the rstatd file
service rstatd {
type = RPC
socket_type = dgram
protocol = udp
server = /usr/local/sbin/rpc.rstatd
wait = yes
user = root
rpc_version = 2-4
disable = no
}
Ensure that the value specified by the server parameter is actually where the rpc.rstatd daemon is installed. If it is not installed in /usr/local/sbin change the parameters above to reflect this configuration.
- Restart the inetd (or xinetd) daemon:
Restarting the inetd daemon
/etc/rc.d/init.d/xinetd restart
- Now rstatd should be running. Running the following command will help verify that rstatd is actually started:
rpcinfo -p localhost
The daemon should be listed several times, which is due to the different RPC versions available for clients to use.
Portmapper service
The rstatd uses the Remote Procedure Call (RPC) system to facilitate connecting to the system portmapper daemon running on a well-known port. The rstatd will use RPC to create a listener that asks the portmapper for a port to use for communication with a remote client. The portmapper selects an unused port and assigns it to the listener.
Performance Tester requires the portmapper daemon to be running on the remote Linux/UNIX system. Run the following command to verify that the daemon is started:
Verifying that the rstatd has started
rpcinfo -p localhost
This command will generate output similar to the following output:
Typical output of rpcinfo command
program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 32768 status 100024 1 tcp 32769 status 100001 2 udp 32770 rstatd 100001 3 udp 32770 rstatd 100001 4 udp 32770 rstatd
The output indicates that the portmapper is operating using the TCP and UDP protocols using port 111, the default port.
If the portmapper daemon is not listed in the output of the rpcinfo command, as shown above, the daemon may not be installed or not configured to run. It is recommended to contact your system administrator and request that the portmapper RPC program be started.
To view the most commonly used RPC program numbers is held in a file called /etc/rpc on each system. The file consists of a series of entries, one per service, such as:
Typical RPC program numbers
portmapper 100000 portmap sunrpc rstatd 100001 rstat rstat_svc rup perfmeter rusersd 100002 rusers nfs 100003 nfsprog ypserv 100004 ypprog mountd 100005 mount showmount ypbind 100007 walld 100008 rwall shutdown yppasswdd 100009 yppasswd etherstatd 100010 etherstat rquotad 100011 rquotaprog quota rquota sprayd 100012 spray 3270_mapper 100013 rje_mapper 100014 selection_svc 100015 selnsvc database_svc 100016
Configuring a performance schedule
To configure resource monitoring using UNIX rstatd monitor, create a new or add an existing location to a given performance schedule, as described in Resource monitoring using IBM Rational Performance Tester: Part 1, Scheduling and reporting.
Enter a target Linux/UNIX host name from which to collect data from and select the UNIX rstatd monitor item from the list of available data sources.
On the Resource page, select the type of data you want to capture. The tree view shows all available performance counters, with a default set of counters pre-selected.
Clearing Show only selected counters allows us to see all available counters. Be selective; monitoring all possible resource data requires substantial amounts of memory. Hover over a counter with your mouse to see details about what that counter measures.
On the Options tab, the time interval properties can be configured:
- Change the Connection information if needed. Typically, you only need to change this information if a firewall is blocking the default port. If the portmapper has be configured to use a different Protocol or Portmapper Port number, this area allows for custom configuration to be applied for the specific target Linux/UNIX host being monitored.
- Enter the Polling Interval (in seconds), for collecting resource data. For example, if you accept the default of 5 seconds, counter information will be collected at 5-second intervals from the specified host during the schedule run.
- Enter the Timeout Interval (in seconds). If the resource monitoring host does not respond within this amount of time during a schedule run, an error is logged.
Once configuration is complete and saved, the resource monitoring location is added to the performance schedule.
