Performance Testing in Production
In this Blog Post we are going to discuss performance testing in production.
Now before you think we have gone mad and lost our minds completely this is not as crazy as it sounds.
Production is an environment that:
- Is sized accurately,
- Has a suitable diversity of data,
- Has the correct data volumes,
- Is on the correct infrastructure.
All the things you spend a significant amount of time getting right in your performance testing environment and that can be difficult to achieve.
Therefore, it seems like the perfect environment to performance test in.
Now a few years ago this would have seemed like madness as testing of any sort was always considered a bad idea in production as it may cause issues with your production application and result in downtime, loss of revenue, loss of reputation and you were almost certainly not going to get approval to do so from your management team. But technology evolves and the way we deliver technology evolves with it and therefore the way we develop, and test technology must change as well.
Performance testing in production is not for all organisations and it does depend on how you deliver software to production but in the right organisation it is a powerful aspect to your performance testing capability. If you have a performance test environment at your disposal that accurately mirrors production in all aspects then this will not be for you as there is no reason to, but if you have not would you run tests on inferior or incorrectly sized infrastructure and attempt to extrapolate their meaning or get an accurate reflection on the correct sized and populated environment.
Let’s explore further.
A few caveats¶
Before we look at performance testing in production, we need to clarify some things.
We are not suggesting that all releases are tested in production and certainly not the initial release. If you are in the process of developing a new application with iterative development cycles you will need to performance test this in more conventional testing environments as you will be running, analysing, debugging and re-executing in line with your development activity.
If you are developing a major upgrade to an application again this might not be a good candidate to performance test in production for the same reasons as above. But if you have a small bug fix or feature being added, or you are removing a feature toggle for code that is in production then why not test it there.
Things to consider¶
Consider your test coverage¶
As we have already suggested production is not a great environment to execute performance testing for significant releases but minor ones where a light touch regression test will suffice should allow you to push code directly to production and execute performance testing there.
If you are going to use production to performance test you need to use application flows that do not leave a trail of data. Stick to processes that stress the application under test while avoiding creating new data or amending existing data. The only possible caveat to this is that it may be possible by working closely with your business teams to create data to allow more complex application flows if the data meets a pre-defined set of characteristics that will allow the business to effectively filter this data from the remainder of production data.
Need a regression pack in production for baseline¶
If you are going to execute tests in production, you need a baseline and your will need to have a regular regression test being executed.
Only executing tests periodically in production will not allow you to form a comparison between test runs as unlike other test environments production is going to change, certainly in terms of data volume and diversity, and possible in terms of infrastructure patches or versions.
Therefore, comparing a performance test run from one execution weeks or event months ago will not prove conclusive in terms of comparison and regression analysis.
Let your production monitoring team know¶
Production will be under a significant level of monitoring and therefore if you are running performance tests in production, whether that is regularly or periodically you need to let your platform support teams know. They may otherwise consider it an attack on production if they see an excessive load that is uncharacteristic and above the load normally expected.
Many monitoring software analyse traffic on changes around the average or mean load and so a significant increase in transactions will certainly trigger monitoring thresholds which can be suppressed for known performance testing periods.
Pick non-peak times and avoid times when batches are being executed¶
You need to consider when you are going to run your performance testing in production as unlike test systems the system will probably have series of batch processes or maintenance routines that run out of hours. Equally you do not want to execute performance tests in production when it is under the greatest level of load from normal business activity.
You need to find a time when you know the system is consistently stable and use this time to run your tests, this will ensure that your performance testing is being carried out again a environment in a known state. It also ensures that comparisons with previous results will be accurate due to the background load on the system being comparable between performance test runs.
Feature toggles¶
Another consideration when running any sort of testing in production, not necessarily performance testing is to have your new code or features hidden behind toggles.
This way you can performance test in production and if the new feature does not meet your non-functional requirements, you can keep it toggled off until a solution to the performance issue is found.
Even if a feature toggle is not expected to be removed for a while for a business reason, maybe it’s a regulatory or mandatory change to your system, then you can at least complete your performance testing safe in the knowledge that when it is toggled on performance will not be an issue.
Conclusion¶
There is often a taboo around performance testing in production but if you take the necessary steps to keep production data safe and to avoid significant load at your businesses peak times there really is no reason not to.
Hopefully this post has provided some insight into practises you need to consider and will give you the confidence to at least promote the idea of performance testing in production.