What makes a realistic load test
Performance testing covers a wide range of different tests. The benefits can vary whether you test a single URL or a complete user journey through the application. It might seem obvious, but to add the most value to your load tests you should make them as realistic as reasonably possible.
What is the point of making a test realistic?¶
Quick and unrealistic load tests often prove useless or even counterproductive. It is often the best way to be overwhelmed with unreliable data.
A realistic load test will allow you to:
- Stress all layers of your application. This way you can assess the performance of all your servers and detect bottlenecks.
- Pinpoint real bottlenecks. For example a simple load test based on a single URL might point to limitations on your front servers when actually the real users won't experience it because the backend is the first to fail.
- Avoid false positives. If you fail to launch realistic tests and encounter a lot of errors, you might stress the CPU of your front servers to handle these errors. The same goes for the network if you don't simulate the load from the right location.
How can I ensure the maximum realism?¶
As obvious as it might seem, prepare your tests! Load testing without preparations is very likely to lead to a false positive or useless test.
Prepare your virtual users¶
As virtual as they are, the virtual user profiles must simulate a real behavior. The first step toward this is to record real user journeys through your application (login + purchase, etc...). You do not want to end up putting load on the wrong components. We have already discussed how to select the functionalities to test in a previous article.
Record and forget¶
A frequently overlooked step is then to make sure your virtual user profiles are working as expected. Most experienced testers know how to do a correlation. But despite this fact a lot of tests are launched with virtual user profiles that do not work properly.
Common methods to ensure correct correlations include:
- Replaying your users and comparing the results.
- Checking the HTTP response codes (4XX and 5XX usually mean bad news).
- Controlling the content of the response to ensure there is no error message.
- Running a unit test and making sure no specific step has 100% error rate.
- Etc...
The importance of data¶
Once this is out of the way, you need to prepare datasets. Simulating thousands of users connecting with the same account will most likely always hit the cache or the same components.
Take for instance a cart on a web store. If you rely on the same user account, you will have concurrency issues when adding products to the cart. But also you actually create only one large cart instead of a lot of smaller ones. This will probably have a different impact on the memory.
In other kind of applications, you might end up queuing requests to the database to update the same record. I have seen many times complaints on database performance because of a wrong dataset.
Add some logic¶
Last, you might want to use branching and conditions to make your user profiles even more realistic. If we stick to the e-shop website example, you should consider that not all users that create a cart will purchase. In fact if you advance further in the purchasing process, usually you see less and less load. This means having a percentage of chance that your user journey finishes on every page. Doing so is important to avoid stressing demanding features like the payment more than real users would do.
Prepare your load policy¶
Defining an accurate load policy can be quite challenging. Hopefully you might be able to get your hands on statistics to help. Google analytics or any equivalent is a very good way to prepare a load policy.
Keep in mind that the load policy must represent standard load but also peaks of activity.
Also, if your application is running 24/24, consider:
- a soak test on a long duration (from 6 to 24 hours)
- a failover test to assess the impact of shortages
Where are my users?¶
To generate a realistic load, you must also consider where you real users come from. I have seen tests run internally pointing to a network issue when in fact the real users where redirected to a CDN thus generating much less network traffic on the infrastructure. Also, if you don't have a responsive website, you might serve a different version of your application to different devices. Add different bandwidth limitations to this cocktail and you will get very different results.
Against which environment?¶
Carefully selecting the environment is also critical to improve the tests realism. But this topic was also covered in a previous blog post.
To sum up¶
As often, preparations are the key to be on the safe side, even for a seasoned tester.
To avoid false positives, your load tests should be as close as possible to a real life event. To achieve this, you must:
- Create realistic virtual user profile
- Prepare the datasets in advance
- Add some logic to your user profiles
- Simulate load from your users real location
- Prepare the test environment