Part 4. Reporting in Rational Performance Tester

 

+

Search Tips   |   Advanced Search

 

  1. Test plan
  2. Report Component: Counters
  3. Types of Reports
  4. How to navigate
  5. Demystify performance reports
  6. Formal Definitions
  7. Performance Report
  8. Overall report
  9. Summary tables
  10. Page performance
  11. Response vs. Time Summary
  12. Page Throughput
  13. Server health
  14. Resources
  15. Page Element Report
  16. Overall
  17. Response time
  18. Page Element
  19. Server Health
  20. Other reports and customizations
  21. ResourcesLearn


Test plan

Task Description
Workload analysis Gather the requirements and come up with test cases to emulate the actual workload.
Test case generation
Server assignment Servers should be sized appropriately to cater to the requirements.
Test recording Start recording the test environment.
Test configuration Start configuring the test environment.
Emulate the workload and playback Pump actual load based on the test cases generated.
Report generation Reports are being generated for analysis purposes. Any application- or system-level bottleneck can be identified easily.
Document bottlenecks Provide feedback to developers on an application bottleneck as early as possible based on the performance analysis reports that you generate.
Document breaking points for an application Document components that break for a particular load, after the initial load was perfect. A use case for this scenario may be workload increase after the initial deployment.


Report Component: Counters

Reports are for the most part made up of counters.

If you double-click test run result counters under All Hosts, that will not bring up a report. These counters in the Performance Test Runs view merely serve as containers. The generic counter can be divided into HTTP, Citrix, and SAP counters.

HTTP counters are composed of the following categories...


Types of Reports

RPT provides application- and resource-level analysis reports. Application analysis reports have each thing to do with application system performance-related issues, such as...

Resource analysis reports relate system resources utilization such as...

Currently, there are three types of resource monitoring data collected:

In RPT, the performance test reports that are provided are HTTP, SAP, and Citrix, of which the latter two require extensions (SAP and Citrix reports will not be discussed in this tutorial).

HTTP performance reports...

Report Category Report Tab Name Description
Performance Report Overall An overview of how test pages are doing overall, including success rate in percentage for...

  • Page Status Code
  • Page Element Status Code
  • Verification Point (VP) Status

Summary A quick summary of three categories of summaries...

  • Run Summary
  • Page Summary
  • Page Element Summary

Page Performance Performance summary rendered in a bar chart detailing the response time for the slowest 10 pages.

Response vs. Time Summary A summary of page performance presented as a line chart, averaging response time for all pages and page elements for a given interval.

Response vs. Time Detail A trend analysis report graphed with each page per line, in which each point represents the response time per interval.

Page Throughput Page hit rate and count per interval, with user load given alongside.

Server Health Summary A summary of system health, with the focus on page and page element health.

Server Health Detail A health detail report featuring 10 pages with the lowest successes. This includes counters such as the attempt and hit count per page.

Resources Resource counter monitored.
Page Element Report Overall Average response time for all page elements for a given interval.

Response vs. Time Summary Average response time per page element for the 10 slowest page elements for a specified interval.

Response vs. Time Detail Average response time for each page element, in detail.

Page Element Throughput A two-graph trend analysis report graphed in a given interval for page element hit rate and user load, respectively.

Server Health Detail Success rate in percentage for the 10 slowest page elements during the test run.


How to navigate

Various performance-related reports are generated and saved automatically. One quick way to gain access to these reports is to view reports via the Performance Test Runs view. By default, this view is available from the bottom left panel (if you have closed it, you can go to menu bar and select...

During performance test runs, the test results are automatically populated in the test administration server for further analysis. Each performance test run results in a line item in the Performance Test Runs view, including the timestamp of the actual run

The simplest way to navigate to the record is to expand the desired performance test run result, and then right-click the All Hosts icon for a list of reporting options.

Reports are divided into categories, and each category contains tabs with unique names.

For example, the first option...

...displays the default report, which is the HTTP Performance Report.

Alternatively, the Performance Report can be obtained by selecting...

Likewise, page element, percentile, and verification point reports can be obtained in similar manner.

There are also the Display Report and Display Transaction Report options.

For example, to display the Transaction Report, you can either select Display Transaction Report directly or Display Report, which lists Transaction Report as one of the report options. Display Report allows you to select from all of the available reports, including custom reports.


Demystify performance reports

This section will explain performance reports (although many of them are self explanatory) by focusing on and explaining the content of each report. In order to pinpoint performance bottlenecks, analyzing the correct reports is essential. You will be viewing the default report presentations in this section; the customizations of these reports are discussed in Part 5 of this series.


Formal Definitions

The following terms are used in this tutorial:


Performance Report

Performance Report is the default report displayed after a test run. It gives a summary of important information (such as run, page, page elements, and transaction summary), together with the details on page health checks (such as response time breakdown and workload trend). As you saw in Table 1, there are nine analysis reports (each with its own tab) that fall under this category.


Overall

The Overall report shown during a test run includes a progress bar that indicates the stages in the test run.

The progress bar messages include...

  1. Initializing Computers
  2. Running
  3. Performing Test Log data transfer
  4. Complete

...and are useful for determing if a long-running test run is still in progress, mitigating the risk that you might mistake a long-running test for a hung process.

This Overall report includes...

The first two bars are always there. The third or fourth bar is only available when verification points are enabled.

The success rate of page and page element codes is affected by such factors as...

Percent Page Status (as the primary request) is usually lower than Percent Page Element Status. This is because a page consists of multiple elements, and a failure on an element constitutes a failure in the primary request.

With verification disabled in the main page, to qualify as a pass for Page Status Code, the primary request return code for a page has to fall between 200 and 300,

With verification enabled in the main page, the main request, by default, inherits a status code of 200 (HTTP V1.1), which you can modify to suit test environment by navigating down the primary request.

Page Verification Point success rate, on the other hand, depends on...


Summary tables

By default, without transaction results being added, you can find three summaries under this tab. In this test run, because the transaction reports were added, four summaries are available.


Run Summary

Active Users Current active users in test run. Meaningful when the test run is in progress. When a test run is completed, this number is reduced to 0.
Completed Users Under normal circumstances, this number increases in proportion to the number being decreased for active users. Upon test run completion, this number should be equal to the Total Users under test.
Elapsed Time [H:M:S] Total run duration displayed in Hour:Minute:Second format. This time indicates the duration from the start to the end of a test run.
Executed Test The test under the test run. This is a quick indicator to show which test is under a project, because many tests can co-exist.
Display Results for Computer By default, this value is set to All Hosts. You can drill down to the performance report for each individual machine.
Run Status Usually Complete upon a test run completion. Particularly useful during a test run (other than using the progress bar) to indicate the progress of a test run. Valid values that may be displayed are Initializing Computers, Running, Transferring data to test log, Stopped, and Complete.
Total Users Total user load. Under normal circumstances, this figure will tally with Completed Users.


Page Summary

Average Response Time for All Pages [ms][for Run]. Sum total of the time taken for each element within a page to respond to a request. This value indicates the average response time in milliseconds for all the pages under test.
Maximum Response Time for All Pages [ms][for Run]. This indicates the maximum response time for all pages.
Minimum Response Time for All Pages [ms][for Run]. This indicates the minimum response time for all pages.
Percent Page VPs Passed [for Run]. This is the same figure provided by the verification point bar in the Overall tab.

Once enabled, this figure shows the total verification point success rate.

Response Time Standard Deviation for All Pages [for Run]. This figure shows the deviation from the mean of the average response time for all pages.
Total Page Attempts [for Run]. The sum total of page attempts made. A page attempt is a request to the server from a primary page, excluding the page elements. This value shows the request without awaiting the response sent from the servers.
Total Page Hits [for Run]. The sum total of page hits. A page hit implies a round-trip of a request (originated from primary pages) and response (from the servers). This value should tally with Total Page Attempts in normal circumstances.
Total Page VPs Failed [for Run]. If set, this value indicates the total number of verification points that failed for primary pages.
Total Page VPs Passed [for Run]. If set, this value indicates the total number of verification points that passed for primary pages.


Page Element Summary

The same information as for Page Summary is available here, except that it is for page elements in total.


Transaction Summary

A transaction is a collection of elements (both primary pages and page elements) that can be gathered for better performance analysis.

Average Elapsed Time for All Transactions [for Run] Average execution time for all of the transactions defined in a test run.
Elapsed Time Standard Deviation for All Transactions [for Run] Deviation from the mean for all transactions.
Maximum Elapsed Time for All Transactions [ms][for Run] Maximum execution time for all transactions.
Minimum Elapsed Time for All Transactions [ms][for Run] Minimum execution time for all transactions.
Total Transactions Completed [for Run] Total number of transactions completed during a test run. This value should tally with the Total Transactions Started under normal circumstances.
Total Transactions Started [for Run] Total number of transactions started during a test run.


Page performance

Because it shows the slowest 10 pages (for an application under test that spans more than 10 pages), this report is best used to identify the slowest primary pages. With one glance, you can immediately identify the pages (using the bar chart) that take the longest time to respond to a test run. This report comes with both bar charts and page response time counters in tabular format. The response time counters (such as minimum, maximum, average, and standard deviation) are provided.

To view response time breakdown data, right-click on the report...

The default display for the response time breakdown statistics are:

The Page Performance report allows you to drill down further to the page element level. It also allows you to drill down from the page element to individual host, application, component, package, class, and method response times for a particular page element.

  1. For example, right-click a bar and select Display Response Time Breakdown Statistics. This brings you to a response time breakdown for page elements.

  2. To see the response time breakdown for a particular page element, select the element by highlighting it in the Page Element Selection wizard, and then click Finish to obtain the breakdown

  3. There is more than one way to drill down to individual host, application, component, package, class, and method response times.

    Another way is to right-click to select Display Host Response Time Breakdown

  4. To navigate to the method and package levels, continue to right-click and use the menu options until you get to method level.


Response vs. Time Summary

The Response vs. Time Summary report is a summary of average response time and response time standard deviation for all pages and page elements. On the left is the average response time for all pages. Each point represents an average response time for a given interval. The default interval is 5 seconds. You can set this interval in the test schedule, under...

To identify any anomalies, look for a spike in the graphs (if there is any).

Response vs. Time Details is a line graph that shows the average page response time for all pages. Each page is represented by a symbol.

For example, in this scenario, the primary page, Trade Order Information is represented by a red cross.

In addition, each point represents an interval for the duration of the test run.

The table at the bottom of the screen, on the other hand, gives page response information (such as minimum, maximum, average, and standard deviation response times, and so on) for all pages.


Page Throughput

This report is two line graphs that show page hit rate and user load, respectively. The left graph shows the page attempt and page hit rate for a given interval. An attempt is the request being sent by a primary page, and a hit is the response from the server upon a request. Naturally, these two lines should stay close to one another, because an attempt will usually correspond to a hit.

If the workload increases without a proper server sizing, however, the number of attempts increases while the number of page hits decreases. When this happens, either scale up the server under test or re-allocate user workload to another server. The right graph displays the active and completed users for a test run represented in intervals. In this scenario, active users were being loaded up around 150 before any user reached completion at 75 seconds. The active users stabilized at around 150, while the completed user line graph shows a healthy linear increase. You can identify an overloaded situation (a combination of high user workload and server capacity) from this graph.


Server health

The Server Health Summary report provides you a summary of server health under test by displaying page and page element related information. These graphs indicate either that workload is too high (parameters such as user think time and start delay time can affect the workload), or that the server lacks the capacity to handle the workload.

The default page counters available in this report are page/page element attempts, hits, and status code successes. The left bar graph shows the page information, and the right graph shows page element information.

The Server Health Detail report is a graph that shows the 10 slowest pages by detailing each page's percent status code success for the test run. A successful status code refers to the HTTP response code passing if a verification point is set for that particular page. In the absence of a verification point, a successful status code falls in the range between 200 and 300.

The information included in the tabular format includes attempts, hits, status code successes count, percent status code success, and so on. In this scenario, the page Welcome to DayTrade has a percent status code success of 96.4. After observing this, you can investigate further by reviewing the test log for this test run.


Resources

This report will only be meaningful if you turn on IBM Tivoli Monitoring, Windows Performance Monitor (PerfMon), or Linux/UNIX rstatd resource monitoring. Otherwise, this report will show a blank page.


Page Element Report


Overall

This line graph displays all page elements versus time. Page elements are, for instance, images, sidebars, buttons, and application scripts that are considered elements of a Web page. The following information is included in the Performance Summary table:

In order to view each element's results in detail, look at the tabs on Response vs. Time Summary, Response vs. Time Detail, Page Element Throughput, and Server Health Detail.



Response time

The Response vs. Time Summary report is a line graph that shows average response time versus time for each page element. Each page element is represented by a symbol that is described in the legend.

You can apply a filter to the result by count, value, or label. When you apply a filter, the result shows just the information that you need.

For example, setting the filter by count to 10 highest response times shows you the top 10 slowest page elements. By knowing the slowest elements, you can take action to lower the response time (for example, use a smaller picture that loads faster).

The following information is included in the Performance Summary table with a count filter of 10 highest (meaning the 10 page elements that took the longest to respond).

The Response vs. Time Detail graph displays all of the page element readings in the run. The following explain the report table attributes:



Page Element Throughput

In this report the left line graph shows the Page Element Hit Rate, and the right line graph shows the User Load. The following are the metrics being used:



Server Health

This bar graph displays the percentage of successes for the page elements in the run. You can apply a filter to the result by count, value, or label. If you apply a filter, the result shows just the information that you need.

For example, setting the filter by count to 10 lowest shows the 10 least successful page elements. These are the Performance Summary table attributes:


Other reports and customizations

This latest part of the series (Part 4) looked at various Performance and Page Element reporting capabilities provided by IBM RPT to generate default performance analysis reports. The next part of the series (Part 5) will show you additional reports, as well as how you can customize and export the reports to suit your needs.


Resources


Learn

  1. RPT product page
  2. instructions for using RPT
  3. Rational software area on developerWorks
  4. IBM developerWorks newsletter
  5. developerWorks Rational zone newsletter
  6. Rational Edge newsletter
  7. Technology bookstore


Get products and technologies

  1. trial version of IBM RPT
  2. a href="http://www.ibm.com/developerworks/downloads/?S_TACT=105AGX01&S_CMP=ART">trial versions of other IBM Rational software
  3. IBM product evaluation versions


Discuss

  1. developerWorks forum