Test results to be collected and verified | Common troubleshooting steps
Verify pass criterion
For all the test cases executed, analyze the results from various angles:
Acceptable failure rate
Failures can happen due to various reasons, including functional defects and performance defects such as deadlocks. Functional defects should ideally not happen during performance testing since if they do, they should not be acceptable. Performance defects, however, may sometimes be acceptable. For example, deadlocks in databases are inevitable. We always try to minimize and eliminate the possibility of having deadlocks in a database, but they are inevitable, depending on the workload at the site. As such, some margin of error, based on your business requirements, in a performance test is generally acceptably.
The page hit failure rate should be less than your business requirements. Another business requirement that you need to track is the scenario failure rate. The page hit failure rate should not result in a scenario failure rate of more than this specification. The difference between the two is that whereas a page hit, such as catalog browsing, may result in a non-functional error, the user should be able to refresh the page and continue with shopping. For example, if there are more than ten page hits per scenario, 0.1% page hit errors and 1% scenario failure rate may be acceptable. Notice that in absolute numbers a scenario is still more resilient to failures than the sum page hit errors.
Acceptable response time
Although the minimum and maximum response times for a page hit and a scenario are important, the average response time is what would generally be tracked. The average response times should also be within your business specifications.
Acceptable concurrency and throughput
Ensure that the throughput targets are met is also important even though it is the level of concurrency (number of virtual users) that seems to be driving the throughput. The level of concurrency is certainly more critical, but the throughput numbers provide you with revenue generation information for the scenario or site design that shoppers interact with. However, scenarios that may be long or non-intuitive may not always translate into orders.
No memory leak or excessive fragmentation
Solving memory problems in WebSphere applications.
No degradation of throughput or response time over time