Analyzing stress test results | Soak, endurance, or reliability testing


Scalability testing

Whereas stress testing involves stressing the system way past the business capacity requirements, the scalability test primarily focuses on performance of the system within and around business capacity requirements. In the case of stress testing after the system has reached its maximum system capacity, response times starts degenerating very significantly, and as such there is not much to be learned from response times. In the scalability test case, response time is very critical, just like the throughput for a given set of test attributes, for example, concurrency, think time, and scenario.

The size of the catalog or the number of hosted stores in the system would also have a big impact on the performance of the system. Thus, more than one data parameter may also be varied from one hurdle to another. In the previous example we have only used users as a parameter for our stress test, but in reality there may be more than one.

Your database size is a major contributor to what throughput or response times your site may be able to provide.

In our previous discussion on stress testing we focused on increasing workload (virtual users and thinktime) for any given scenario. For example, we focussed on the test attributes only. In scalability testing we follow the same tactic, plus we test for different control attributes.

This is an arbitrary distinction since different data sets, hardware setups, and so on, should be stressed as well. You may decide to combine both your stress and scalability testing in one series of test cases, starting from low load to peak load, and then eventually to the breaking point load. However, if the thinktime in your stress test is not realistic, then you would have the same additional work of scaling the test results to your production environment size before any meaningful decision could be made from them, as discussed in 23.3.1, Testing for throughput.

The reason that we find this distinction useful is that, depending your site, for a given deployment you may need more or a different level of focus on either stress or scalability. By lumping the two together you may lose the focus and even end up doing more testing than you actually require.

To test for valid response times the system needs to operate within or at its maximum system capacity. The site can still be operating way past the expected peak workload as defined by your business. Some sample business capacity numbers are discussed in Expected peak capacity versus maximum business capacity.