What is the plan?
If you are a tester, chances are you already know what a test plan is. If not, the short definition would be : "A specification document of your load test campaign".
This document is very close to the functional test plan in its structure because both share some common ground. The question I'm often asking myself when preparing a load test is if this document should be used at all.
Don't get me wrong, the thinking behind it is mandatory. Otherwise your test campaign might not be very effective. But what about the actual document a tester usually produces? For years, I have been a tester working for different customers every week or every two weeks. I have to admit that I don't rely on a test plan half of the time. Even if you don't switch context so often, I believe there is some common ground depending on the project length and complexity.
Short tests¶
Come to think of it, load testing is often conducted during short timeframes (a week or less). In that situation spending a whole day creating a documentation of the campaign is probably a waste of time.
A load tester is bound to encounter these very short test campaigns, their purpose can vary :
- To get external approval and/or reassure stake holders.
- In case of external development, to assess performance during the application warranty period.
- Part of an agile dev/testing process.
In these situations, I find it difficult to tell the customer he has to pay an additional day, which might represent 20 to 25% of the load testing budget, just to create a test plan.
Agile testing¶
As development method tend to be more and more agile, testing has to adapt. Preparing a full blown test plan for every test iteration does not seem appropriate anymore. Writing one for the first iteration might seem a good idea but as the testing strategy will evolve can you spare the time to update it? If not, it wont serve as a proper documentation, so what's the point of this document anyway?
Longer tests¶
Of course sometimes, the strategy and objectives are much more complex and it feels more natural to write them down in a test plan. Well then you face another issue. In my experience, very few people will go through the test plan. Most customers I worked for were looking for minimum involvement. That means they trust you with the test methodology and don't want to have to validate your test strategy.
Plus if your communication skills are good enough, chances are that they don't need a test plan to understand the testing strategy. In that case, if you create one it is very unlikely to be read by anyone except you.
Lastly, if you have run a load test campaign before, you know that it almost never happens according to the plan. After running the first test you usually have to switch plans to make the most out of the remaining time. In this case you will probably not have any time to keep the test plan up to date.
Why would I do a test plan then?¶
Don't get me wrong, I'm not trying to say that a test plan is useless in every situation. Of course, I'm as lazy as anyone and the less I have to work on documentation the better I feel. But that is not why I recommend not to do a test plan. The main reasons I personally do one are :
- The customer is new to performance testing or has no clear vision of the tests.
- The load tests are prone to be repeated in the future, and even more if it is by someone else.
- The customer wants everything to be documented.
Ok, no test plan then¶
If you don't do a test plan, I still suggest you setup a meeting with all the interested parties :
- Functional experts,
- Infrastructure experts,
- Project leader.
After this meeting, send a debriefing e-mail which will act as a "field" test plan. That way you keep record of this brainstorming meeting.
So why is the test plan recommended so much?¶
First the test plan is often made to protect the tester from a too demanding customer. You write down the scope of the test campaign and stick to it. If anything new is to be added to the campaign you can go back to the test plan and consider it as an extension or out of the scope.
Then a test plan can be used to explain your testing strategy. I find it more efficient to meet up with any interested parties instead but if you work remotely the test plan will do the job just fine.
Another interesting use that I mentioned earlier is toward industrialization. When working in a large team the test plan will help keep track of previously run campaigns. It can be reused for new campaigns and to make sure the results are comparable. But in that case you have to be very careful to update the test plan during your campaign.
To sum up¶
- Consider the duration of the campaign, for very short engagements it is probably better to go for a "field" test plan.
- For longer campaigns,consider creating a test plan for documentation or repeatability purposes.
- While testing as part of an agile process, the test plan is very likely to slow you down if you have to keep it up to date.
- Test plans are important toward industrialization but when it comes to communicating the campaign details and objectives, a meeting along with a debrief e-mail can be efficient as well.