Testing for concurrency | Scalability testing
Analyzing stress test results
For each test scenario executed:
- For peak workload expected and minimum business capacity required, the error rate must be less than the acceptable error margin, as defined by your business requirement. For example, the acceptable error margin may include page errors less than 0.1%, resulting in less than 1% failed scenarios.
- Concurrency, throughput targets should be met or exceeded for both peak workload expected as well as for minimum peak business capacity required.
- Response time targets should be met or beat for both peak workload expected as well as for minimum peak business capacity required.
- For excess capacity past minimum business capacity although response time is allowed to go past the maximum requirement, the acceptable error rate should still conform to your business requirements.
If any of these passing criterion are not met then it should be considered a defect. If error rate is the cause of the test case failure then the defective code needs investigation.
If overall low performance is a concern then we need to go through the iterative process of performance tuning and re-executing the test case, as shown in Figure 23-1.
Last but not the least, if the concern is memory leak, core dump, or gradual throughput degradation, then refer to Chapter 24, Analyzing test results and solving performance problems.