Execution conditions
By execution conditions we mean understanding and controlling the execution of benchmarking tests so that we know the initial conditions of the test and can repeat the tests. Here are two sets of best practices to help you manage test execution.
Test repeatability
| To ensure that a test can be repeated, we should back up the environment prior to the test run. Database and configuration are the two most important parts of the environment. |
|
| We should run comparison-type tests (for example, WebSphere Commerce with and without DynaCache) in the same environment. |
Constancy of workload
|
| We should keep the number of concurrent users constant across comparison-type tests to ensure the workload remains the same. |
|
| Think time between two hits has a significant impact on workload: longer think time means lower workload and vice versa. To ensure the workload remains the same, we should keep think time constant across comparison-type tests as well. |
|
| Whether DynaCache has been preloaded, JSP pages precompiled, and the database warmed up also impact the actual workload on the system. In benchmark-creation tests with DynaCache, we should make sure DynaCache is fully loaded and stabilized, JSP pages precompiled, and the database warmed up. Typically, allow half an hour warm up and a one hour run. |
|
| Make two or three runs to confirm the consistency of the results. |