How do I define my user profiles?
One of the questions I hear the most from newcomers preparing load tests is about defining proper virtual user profiles.
If you are familiar with testing, you probably heard about the 80/20 rule, also know as the Pareto principle. Apart from impressing the ladies at parties (you know, probably), this principle states that roughly 80% of the effect (performance) comes from 20% of the cause (functionalities in our case). This principle is widely used when selecting the functionalities to test and avoid too much and/or too long user profiles.
Performance testing is not functional testing¶
Ok, I know, thanks captain obvious.
But still I see a lot of testers creating more than 10 user profiles to test the performance of simple applications. You must keep in mind that these two types of tests differ in terms of objectives, but also in terms of preparation cost.
Yes, a performance test first objective is usually to improve performance. Well this is not always true, but that's another debate.
If we run a test to assess the performance of the servers, what matters more than anything? To generate a realistic load to the server !
You can measure it in terms of hits/s or functionalities called, or even web service calls. What's important to understand is that two different user profiles might actually generate the exact same load to the server. On top of that, a performance use case is already more complex to design than a functional one so why not keep it simple?
Preparation and costs.¶
A functional user profile should be rather quick to record but costly to replay because you simulate a client for every user (browser or heavy client). The memory cost for this can be important when generating more than one user but it's perfectly fine for a functional test.
On the other hand, a performance test virtual user is made to be launched in large numbers. It is usually more complex to design because you work with the traffic between client and server. If there is anything happening on the client side, you have to reproduce it yourself (extracting a session ID, reprocess data in a different format, etc...). This makes it more difficult to create but lighter at replay because you only reproduce traffic. That's why you want to keep the number of virtual user profiles to a minimum whenever possible. Let's see how we can achieve that.
Avoid redundancy¶
The first thing to consider in redundancy. When possible, avoid using the same functionality in several virtual user profiles. For instance, creating a virtual user for each product on your web site does not make any sense. You can create one that will select a random product instead.
Correlation will help greatly against redundancy. Keep in mind that anything in the server responses can be extracted. This way you can create smarter use cases.
Limit the number of virtual user profiles¶
If you are creating more than 5 virtual user profiles, you are either dealing with a very complex application or you are making a mistake. The 80/20 rule is just a way of saying that covering 100% of what our users do would not be cost effective. We have seen that it takes time to prepare. But it will also be difficult to maintain!
I have seen many clients and prospects struggling to create and maintain tens of virtual user profiles. In the end with such amount of work, you will probably end up doing a mistake. And a poorly working virtual user profile is a lot worse than no user profile. It can lead to wrong decisions which might be costly. On top of that, you must consider the chance that someone different will run the tests next time. Even with a proper documentation on all the scenarios (which would also take some time) you have an increased risk of error.
In a nutshell :¶
- Consider user profiles in terms of load to the server rather than functionalities
- Avoid redundancy
- Keep a low number of virtual user profiles (1 to 5) to allow for faster preparation, maintenance and lower the risk of error.