What makes a good performance tester
You have probably read countless articles about what a good tester is. Of course this is a tricky question and there is no perfect answer. Performance testing requires a mix of technical, organisational and communication skills.
I have seen good technical testers fail to deliver the right message even if they identified the source of all issues.
On the other hand, I have seen performance testers with low technical skills perform very well thanks to their communication.
There is no perfect mix of expertise I would recommend but instead, let's see what the testing challenges are.
Technical skills¶
Any performance tester should have basic knowledge of the protocol under test and the way the application works. This is critical to get a realistic test campaign done properly. Take correlation for instance, most virtual user profiles will require you to correlate data received in the server response with data sent in the next requests.
Not being able to do this means launching users that will fail on certain steps. Because of that your test will generate a lot of errors on the front servers and nothing on the back end servers.
In that situation, your analysis is very likely to point out non existing issues. You would end up spending time running tests to waste time fixing non issues. That's why technical skills are so important in achieving a realistic performance test.
Communication¶
You have probably guessed it from the introduction, but my opinion is that communication is the key!
Even if performance testing is often referred as a technical test, good communication is what will make your tests outstanding. I already mentioned the impact of communication on a load test report in a previous article.
But the performance tester is also in the middle of all the project team, during a typical campaign you will ask:
- Functional experts to describe the use cases.
- Project managers to give SLAs and/or take decisions.
- Developers for scripting technical support and virtual users preparation.
- Infrastructure specialists to prepare the environment and infrastructure monitoring.
On top of this, remember that you work on a tight schedule. Performance testing is usually done in a matter of weeks. This will lead to a variety of challenges to be able to synchronize, get preparations done and communicate with everyone.
Synchronization¶
A big challenge for a performance tester is to synchronize all the teams involved in a performance test. I once found myself working with 4 different companies for a single load test.
On most campaigns I did these past years:
- the hardware is outsourced,
- the software development is outsourced,
- both might rely on 3rd party products to get the job done.
In this situation the first challenge will not be your technical skills but your capacity to synchronize everyone. You will have to work with people you have little to no authority over and get them to trust you.
External expertise¶
The first thing I tell to all the people I train is:
"A performance tester is expert on performance tests and testing methodology, not on every technology out there on the market".
It would take more than a lifetime to get expertise on every technology you might one day have to test. You might of course have an idea of how to optimize let's say a tomcat server because you encountered many during your career. It's good but that won't replace the expertise of a DBA for instance.
Instead, you should ask for expertise when preparing the test campaign. Usually the company asking for the load test has the expertise on the components they use. This adds another layer of synchronization to your campaign.
Tests preparation¶
Most of the time your contacts won't understand what you are asking or why you are asking it. It is then up to you to explain the purpose of your demands. For instance, requesting dataset for your scenarios is fine. But most of your contacts will have no idea of the required dataset to run a test. In this situation, you must check the user journey and request all the specific data you need (login, password, etc...).
What's the worst that could happen?¶
If you still think that technical skills only will grant you success, let me tell you a story.
This happened to a fellow tester in one of my previous jobs. He did a performance test campaign which showed pretty bad results in regards to expectations. He had prepared everything correctly, the user profiles even comprised a script to reproduce an asynchronous behavior. He wrote down in his report that the application could not handle the load. But one of the first graphs showing in his report was about how many HTTP connections were active before the breakdown.
This was misinterpreted by the project manager for number of concurrent users. As you might know the number of HTTP connections on a web server will be a lot higher than the actual number of users behind them. In the end the application went live because the project manager thought it would handle the load. From there as you can guess things went ugly...
Of course the project manager should have read the entire report but the deadline was close, time was scarce, you know how it is. In the end this communication mishap caused the whole test campaign to be useless. The tester did his job by telling the customer in the report, but he might have made it clearer that the application was not working fine under load.
Since this day, I always start my reports with the conclusion to give a go/no-go before anything else.
The ugly baby syndrome¶
The most challenging time in a test campaign is when you find a bottleneck. A bottleneck usually means a problem to be fixed. And being in a position to tell people what they should fix is not easy.
This is what David Whalen calls the ugly baby syndrome. Of course nobody is happy to be told their offspring is not perfect but as a tester it's your job. That is where the way you interact and communicate with the rest of the team will be the most important.
To conclude¶
Basic technical skills are required to run a realistic load test BUT communication will be the key to get the most out of your test results because:
- You have to work with the whole project team on a short schedule.
- Some of the team might be from external companies you have no authority over.
- You can't have all the expertise by yourself.
- You might have a hard time getting the issues you raised considered by the project team.