Volume Testing
In this post we are going to look at the importance of volume testing. We are going to consider how this type of non-functional test can have a significant impact on your size and scale your production environment based on evidence this test provides. We are going to look at some real-world examples of how getting your volume testing correct will ensure that your environments are not oversized or undersized. Both these scenarios have a cost which is both financial and potentially reputational.
Definition of volume testing¶
From Wikipedia.
Volume testing belongs to the group of non-functional tests, which are a group of tests often misunderstood and/or used interchangeably. Volume testing refers to testing a software application with a certain amount of data to assert the system performance with a certain amount of data in the database. Volume testing is regarded by some as a type of capacity testing, and is often deemed necessary as other types of tests normally don't use large amounts of data, but rather typically use small amounts of data.
We use this test to understand the impact of large data volumes on your application under test. Larger data volumes in the database can:
- Increase the query time of your SQL statement which impact response times,
- Impact the amount of CPU or memory your database uses and therefore you can use this test to size this component,
- Impact the size of responses to end user and API requests, with search requests being an example, this impacts application CPU and Memory.
A volume test looks to not only understand performance of the SQL with an indicative sized database. But also looks to understand how much CPU and Memory is required across you full stack to support peak volumes of load and concurrency. The results you gather from this type of non-functional testing is important in determining how you application needs to be sized in production to support future data volumes.
Setting the scene¶
To run a volume test there are a couple of things that must have already been completed. - You need to have defined your non-functional requirements, so you understand your response times and load levels, - You need to have run a performance test at these levels with a database not at full production volumes and have a set of results, - You need to understand go-live volumes and the expected growth pattern for your production system.
It is of course conceivable that if this is a new system the database tables will be relatively empty when the application first goes live. Which means that you may already have a set of results at go-live volumes as this would have been the performance test you have run.
To demonstrate the principles of a volume test we are going to look at this is in a theoretical way rather than testing a dummy application. The theory is based on our experience as performance tester and will give you enough knowledge to go and create your own. Volume testing is never the same for any two applications therefore even if we did test against a real live application our findings will in no way reflect those you get from your applications.
Defining our requirements¶
Before running any performance test, it is important to gather a set of non-functional requirements. These define the levels of load and concurrency you should be running your performance tests against. Make sure that you also define data volumes for go-live and predicted growth. Clearly data volumes analysis for new applications is not an exact science and is sometimes guesswork, we will demonstrate later in this post how we legislate for this.
Let’s build some dummy requirements to help with our post, these are very simple.
First business functions:
| 95th percentile | Transactions per minute | Database entries at go live | Expected growth per year | |
|---|---|---|---|---|
| Business function A | 1000 ms | 200 | 20,000 | 10,000 |
| Business function B | 1000 ms | 400 | 40,000 | 20,000 |
| Business function C | 1000 ms | 800 | 26,000 | 13,000 |
Then Servers:
| Max CPU under load | Max Memory under load | |
|---|---|---|
| Server A | 50% | 80% |
| Database A | 50% | 80% |
Creating our baseline¶
Before we start, we need a baseline. This baseline is normally the results of your performance tests under peak load and concurrency that have met your non-functional requirements. You will naturally have some data in the database as you would have been building your tests and executing these to satisfy your performance testing. It is unlikely, but not impossible, at this point that you would have a production sized set of data in your database tables.
If you do not have production sized data in your database tables then you have what we will call Baseline Zero, this is a made-up label for the purposes of this post. A performance test with go-live production data in your database is your first volume test.
If you do have go-live production data in your database, we will call this Baseline One. You may have production sized data in your database for one of two reasons:
- Your development teams have built a data migration solution already and this has been run in your performance test environment.
- Your application is new and is not replacing anything, therefore you will at go-live have a relatively empty database.
Depending on whether you have Baseline Zero or Baseline One defines what you do next.
If you have a Baseline One then you want to make sure you have at least this information:
- Response times of all transactions.
- Count of all transactions executed.
- CPU metrics from all servers.
- Memory metrics from all servers.
- SQL query durations.
- Database volume count.
This is because you want to measure all future volume tests against this baseline. If you do not have a Baseline One but have a Baseline Zero then you are going to need to get your performance test environment loaded with data to reach go-live volumes. Once you have achieved these volumes in your test environment you then need to re-run your test and get the above information, and this becomes your first volume test. We will discuss how you could load production data into your performance test environment in the next section.
Data loading strategy¶
- If you do not have production sized data in your performance test environment database, then to run your first volume test to get your Baseline One as we have termed it, then you need to find a way of loading data.
- If you are needing to load data to reach go-live volumes, then you must have a strategy to do this in your production environment as part of go-live.
- If this is the case, then you could use this mechanism to load data for your performance tests. This has an additional advantage of also being a good test for the migration tool. It is possible that your migration tools are not yet built or the mechanism for migration data is you to be finalised.
- If you are looking to run your volume tests early in the delivery lifecycle, which is a huge benefit to the programme from a performance perspective, then it is understandable why your migration tool may not be available.
- If you want to learn more about shift-left performance testing then there are articles on test strategy and running JMeter from pipelines.
An alternative to data loading to achieve your performance data volume requirements is to use your performance tests to generate data. During execution of your peak volume or soak tests you will generate persistent data in your database. If you count your database table rows and then execute a load test or soak test and then count the rows afterwards you get your net gain of database entries for a test.
You can then execute your load tests any number of times until you reach your target. A limitation of this approach is that your performance tests probably will not cover all business functionality as you will have risk assessed the performance tests you need to write and not all business functionality will be required. I would suggest that if most of your database tables are populated with go-live production volumes then this is good enough.
Volume test¶
As mentioned above, once you have executed your first performance test with indicative go-live production data volumes in your database you have run your first volume test.
With the information you have gathered from your first volume test you need to ensure that your non-functional requirements have been met. If they have not, then you need to understand why and resolve these volume related issues. If for example your application has slowed down to the point that they no longer meet your response time requirements, then the data volumes have impacted performance. At this point you need to investigate SQL query times and your CPU and memory consumption across all your technology stack. You can then re-run your volume tests as fixes are applied until all non-functional requirements are met.
A lot of the time volume related performance issues are down to poor SQL or missing indexes, or lack of CPU and / or Memory. Once you have got to the point where you are happy with performance in your volume tests then you will have also defined your go-live configuration for Infrastructure CPU and Memory in your production environment. Once you are happy with performance under go-live data volume conditions you then need to push your data volumes above those of go-live.
Future volume testing¶
You will have noticed in our dummy requirements gathering section of this blog post that we have requirements around volume growth. This is an important set of requirements to capture as we need to understand the impact of data volume growth on the performance of our application under test.
Your data volumes will start to grow from the minute your application becomes a production system, and it will continue to grow over time. The best approach to determine the impact is to build incrementally to one year’s growth. The reasons for this are firstly you may not be able to reach the levels of application performance you need with a full year’s growth of data. But you might be able to achieve six months growth of data and still meet your performance requirements. This gives you time to investigate the reasons why this the case and means that you can go-live if you want. You then have six months to address the performance issues. Another reason for increment steps is that you can determine how performance regresses, if it does at all, as volume grow.
Let’s look a theoretical example where we run a series of volume tests with monthly increments of data volumes.
Before we look at the response times and start to make sense of the results, we need to consider the volumes.
- Axis X is months, with zero being our Baseline One.
- Axis Y is time in milliseconds.
Our non-functional requirements state that:
| Database entries at go live | Expected growth per year | |
|---|---|---|
| Business function A | 20,000 | 10,000 |
| Business function B | 40,000 | 20,000 |
| Business function C | 26,000 | 13,000 |
So, entry 0 on the x-axis is go-live volumes, these are effectively our Baseline One results where we have run our performance test with data volumes indicative of those identified above. We can see that we have also identified in our non-functional requirements an annual growth rate.
- Business Function A data volumes must be increased by 10,000 / 12 = 833 records for each monthly increment.
- Business Function B data volumes must be increased by 20,000 / 12 = 1,666 records for each monthly increment.
- Business Function A data volumes must be increased by 13,000 / 12 = 1,083 records for each monthly increment.
Between each test you will need to use your data loading strategy to increase volumes.
Our non-functional requirements state that each of our business functions must respond within 1000ms. If we look at the results, we can see:
- Business Transaction A meets its response time requirement until month 8 when it exceeds this. The response time of Business Transaction A at month 10 then increases significantly and continues to increase for the final 2 months.
- Business Transaction B meets its response time requirement until month 5 when it exceeds this. The response time of Business Transaction B then remains relatively stable as volumes increase further.
- Business Transaction C meets its response time requirement until month 7 when it exceeds this. The response time of Business Transaction C grow steadily as the volumes increase.
We have also built a theoretical graph of CPU and Memory. In your real-world testing you would have one of these for each server in your application stack, but for the purposes of this post this will suffice.
- Axis X is again months.
- Axis Y is a % value.
If we consider our non-functional requirements.
| Max CPU under load | Max Memory under load | |
|---|---|---|
| Server A | 50% | 80% |
| Database A | 50% | 80% |
We can see that the amount of Memory we have on our server is correct and is enough to support our future volumes. We see at month 7 we start to exceed our CPU requirement of 50%. Whilst this data is fabricated it does show the sort of information you can get from your volume tests and why they are important. With the information we have extracted from our tests we can make a clear recommendation to the programme about not only performance of the application at go-live but also how we can expect performance to change over time. You can keep these results and re-run your tests should application changes be made to understand if improvements can be seen.
Archiving and weeding¶
Before we finish it is important to touch on archiving and weeding. This is the process of regularly cleaning unwanted data from your database. If you have transactional data that does not need to persist then you should make sure your development teams have a strategy to remove this periodically.
This is sometimes an afterthought in application development but by understanding that it could improve the performance of your database it is worth talking to your development teams about. If they do implement a process to clear unwanted data, you can again execute your test and compare the results to see if this improves the results.
Conclusion¶
Volumes tests are something that you should always consider when designing your performance testing strategy. Making sure that your application performs as data volumes grow is important as it makes no sense to spend time running performance tests to make sure your application meets their non-functional requirements only for performance to regress after go-live.
This type of performance test can provide you with confidence that your application will continue to perform after go-live and as data volumes increase.