The basics of Simple Performance Testing
You don’t have to performance test if you are not bothered about the performance of your application; this is the only reason, if you are bothered and you want to make sure your customers get the best possible experience then it is wise to do some.
You have probably experienced poor application performance if you do any form of online shopping, many web sites are unable to handle the volumes of load and concurrency they see on their eCommerce platforms at peak times sure as Black Friday or Christmas.
Some even have to introduce a queuing system to throttle the load on their systems which leads to customer complaints and a poor reputation.
In this Blog Post we are going to look at some of the things you can do, that are not overly complicated, to reduce the risk of poor performance on your applications; basically, the simplest approach to performance testing we can think of.
Clearly, we would always recommend you invest a fair proportion of your QA effort to performance testing but if that is not something you wish to do then these simple things could be the difference between an application that can cope with high seasonal demands and one that does not.
This is therefore our guide to a simple performance test.
Get an accurate measure of seasonal load from your business¶
This is a straightforward thing to do and if you are determined to only do the least amount of performance testing then you need to find out accurate volumes of load and concurrency.
It is important to understand how the application under test will be used once in a production environment, most applications, if not all, are seasonal and peak volumes of load coincide with an event or time of year.
We have spoken about eCommerce applications having greater demands at Christmas and Black Friday, but all sorts of applications have peaks; for example, insurance companies see higher demands at registration plate change time and holiday insurance demands go up in the summer.
Inside of these busy times there will also be peak demands during the online day and these are important to understand, knowing that your application needs to handle 5,000 users a day and 200,000 transactions a day is not enough you need to understand if these transactions are focussed at a point in the day.
If we perform a quick calculation:
200,000 transactions / 24 hours = 8,333 transactions per hour
8,333 transactions per hour / 60 minutes = 140 transactions per minute
We may then build a test that runs a 140 transactions per minute and think we have covered our peak volumes.
What if most of these transactions happen in the evening when people have more free time and are back from work or at lunchtime when people are on a break from work, suddenly, our load profile looks different.
If we have determined that most business transactions are concentrated at a small number of times of the day that total eight hours then our calculation changes to:
200,000 transactions / 8 hours = 25,000 transactions per hour
25,000 transactions per hour / 60 minutes = 420 transactions per minute
You can see that the difference is significant with our transaction rate at peak times three times higher in the second calculation than in the first calculation.
These examples are extreme, but the point is to speak to the business, find out what happened at peak times last year, look at the production database and run queries against it to determine at what point the application will be under the greatest levels of load and test based on these volumes.
You may need to add a small percentage increase to these volumes to legislate for seasonal growth or increased volumes based on the software you are developing, but a sensible conversation with the business will help with this.
Before we move on, too much load is just as problematic as too little load, if you arbitrarily place thousands of transactions a minute on your application, and it is unable to handle this level of load, you may waste time fixing a problem that will never occur.
Requirements gathering¶
This exercise helps shape your testing and will help you focus in on high volume, high revenue generating flows through your application.
Again, speak with the business and find out what these are and build your performance testing requirements from these.
Requirements can be simple and do not need to be overcomplicated and to combine a set of user journeys with an accurate volume of load can be a powerful combination.
Find the high volume, high revenue business journeys and focus on these as this is where most of your business is focussed.
It is also worth discussing these high revenue business flows with your developers as they may inform you that they cover common code that is used in many areas of your application under test and this with further increase the scope of your simple tests.
Simple tests make a difference¶
Some simple tests can make a difference when it comes to performance testing and whilst we have already stated that really a larger investment in your applications performance should be something you aspire to if you only write a few you will see benefits. You have your core functionality identified as part of your simple requirements gathering and you have got an accurate measure of load. With these simple pieces of information you can write simple and well targeted tests that can provide you with confidence that your application will at least be able to support your high volume, revenue generating functionality.
It is a common for performance testing suits to grow sometime to a level where their coverage is huge and this is not always a good thing as maintenance effort grows and the time taken for debug issues can consume most of your time leaving less time for analysis of your tests against your application.
A relatively simple set of tests is therefore not always a bad thing if they are well targeted.
If you are favouring a simple approach to your performance testing, then your scripts must be robust and should be written in a modular fashion to support expansion of your test suit should you require it later in the process.
There is a good example of the concepts of modularisation here.
Transparency of testing¶
Transparency of your testing is always necessary regardless of whether you are adopting a simple approach to your performance testing or a more complex one, what differs is that under the conditions of simple performance testing you are deliberately keeping your coverage to a relatively small subset of your applications functionality and you may need to justify this.
By clearly stating what the scope of your tests are upfront helps with the sign off process at the end of testing and this is especially true should your coverage be limited to the core functionality.
Share the effort¶
Don’t make it purely a QA asset, by expanding the visibility of your testing may encourage further development of your performance tests by other members of the development teams. This way your simple performance tests naturally expand and grow with little or no input from you which if a simple approach is your ambition then this is a good way to increase coverage.
You could argue that in an Agile Methodology everyone is responsible for Quality and therefore if this is the way your organisation is approaching development then you should be sharing the effort anyway regardless of whether you are keeping performance testing simple or have decided to adopt a complex approach.
Show the benefits¶
You will clearly have a reason for wanting to adopt a simple approach and this may be a time constraint, resource constraint or technology constraint but regardless of this showing the benefits of your relatively simple performance testing along with the information you have from your requirements and volume gathering exercise will provide confidence that performance of your application has at least been considered.
Conclusion¶
The bare minimum is better than nothing and may help you uncover some of the more obvious performance issues, we cannot recommend this as an approach but if you do not have the time, resources or desire to do more this approach will help.
The simplest performance test we can think of consists of:
- Understand the load profiles,
- Gather requirements to focus on a small number of tests,
- Build these simple tests using a robust and modular approach,
- Look to share the effort and clearly show the benefits.
And it will be enough to give you some level of confidence that your application performs and provide enough information to key stakeholders and application owners that the application will perform when faced will seasonal peak volumes.
As we have stated before simple performance testing is always better than none.