Performance Regression Testing
Regression Testing, as all Quality Assurance professionals know, is ensuring that previously developed and tested software continues to operate after a change.
Performance Regression being a subset of regression testing as a discipline is therefore ensuring that previously developed and tested continues to meet its performance criteria after a change.
There are subtle differences in the way that performance regression testing is approached when compared with functional regression testing, and we will look to explore these in this Blog Post.
A suitable performance regression strategy can provide huge benefits to your organisation, and we will look that these benefits as well as how to accomplish them.
Once you have a performance regression test suite it is important that it constantly evolves and is not left to become outdated and eventually obsolete, it must be maintained as we will discuss this also, but the benefits of a well-maintained regression pack will exceed the effort required to maintain them.
Frequency of Regression Testing¶
We should not be thinking of regression in terms of functional regression testing which is normally run when there is a change to the application and subsequently run as part of core functional coverage as part of the release process.
Performance regression like anything that is automated can and should be run much more regularly even if there is no change to the application being developed.
All performance regression should be run at least weekly, when outside of a development cycle, which can clearly be increased if needed. If you are in a development cycle, then regression should be running daily if not after every build.
The benefits section below will expand on when the regression tests should be run to maximise their benefits.
Benefits of Regression Testing¶
Below is a list of the benefits that well thought out performance regression tests can bring:
Regular gauge of performance¶
If you are outside of a development cycle with your application, then a weekly execution of your performance regression tests will be more than enough. It will give you a set of timing point for all tests which you can measure against your requirements, and it gives you a set of known results for your tests at a point in time.
So that if for example you were to start a new development cycle you have a current set of results to use as the benchmark of performance before any new development activity begins.
Historical results for Trend Analysis¶
This is an extension of the regular execution outlined above where this constant execution can be used to plot an historical view of performance. If for example you spot that your regular performance regression test execution outside of a development cycle starts to regress, then it may be due to your test environment or volume of data or any number of external factors.
By looking at response time trends you can spot if there is a sudden regression in performance and work to determine its cause and if it is data related, for example, then you can ensure any issues are rectified well in advance of any planned development that might impact your ability to start testing.
Test Environmental health checks¶
I have touched on this already, but it is worth reinforcing, regular performance regression test execution does perform a health check on your test environment.
If environments are left unattended between development cycles they can sometimes take a while to recover and if you have project deadlines for performance testing that you need to meet, then having an environment that you know is operation will save you time and effort at the start of the development activity.
There is no reason that your performance regression tests could not be run across multiple environment on a regular basis to make sure that all test environments remain stable.
Support functional test regression¶
This benefit is realised during development cycles when both performance testing and functional testing are running in parallel.
If your performance regression is running after every build or running overnight then you are covering some of the journeys that the functional teams will want to execute, and this may help them in their testing cycles.
Production health checks¶
If your company policy allows and your test are non-intrusive from a data perspective, then performance test execution in production can be useful, and using automation so you have a consistent business flow and consistent data reduces the risk.
The performance regression pack, or a subset, can be used to fulfill this requirement should you need it.
Support release process testing¶
If you are working in a DevOps Agile way, then you will be familiar with the concepts of regular releases to production and of functional tests being written in code to support this process. Performance testing in the delivery pipelines should also support these agile ways of working by having the criteria for code release being successful functional and non-functional test execution results.
The performance regression tests you build become the foundation for this.
Check monitoring software updates and changes¶
Production monitoring software needs to be tested to make sure it does alert when thresholds are exceeded, and the best way to test this is to use your performance tests.
Once configured and operational the best way to test any changes to the configuration of the tool or a new vendor release could be to use the performance regression tests, albeit at a higher load, to check that threshold alerts are still working.
Run by development teams as part of release pipelines¶
Expanding on their uses in a DevOps environment it is possible that the development teams can execute the performance regression tests as part of their development pipelines and gauge if their code changes have had any impact on the application performance.
If performance does regress, then the developers know exactly which piece of code they have changed that caused the issues making fixing performance issues simple and efficient.
Ensuring performance tests are kept up to date¶
One of the biggest mysteries in performance testing, and maybe automation in general, is why tests that have not been executed for a while always fail to work the first time they are re-run with seemingly no change to the code or environment.
The regular execution of your tests in the form of regression makes sure that when you are required to move into a development cycle and run tests against a changing code-base that they work from the start.
Performance regression in context¶
While we have discussed the benefits of regression it is important to understand that it must be maintained and added to as more functionality is developed and maybe some existing tests removed. The critical aspect of a regression pack, whether performance or functional, is that it is current and reflective of the way the application under test is used in production.
As we have discussed there are many benefits to running these tests, and some of the ways in which they can be used, especially in a DevOps framework require the tests to be completely stable and robust as well as reflective of recent changes to the application because their success or failure will determine if a code release is pushed to production or at the very least higher environments for further testing.
Regression maintenance is essential, and a regression test should not be something that can just be left unattended and rolled out when required. Its regular execution, maintenance and growth require effort but will provide significant benefits to your organisation.
What should be included¶
Your regression tests should be core functionality, high volume, revenue generating business flows, this is not to say that you could not include all your performance tests but if you do want to reduce them down because of the sheer number you have then this is a useful way to approach what you could remove.
If your test suite is robust and relatively maintenance free, then execution of the whole test pack would ensure all the benefits we have already discussed around regular execution and a constant awareness of your performance tests will apply to all your tests rather than a subset.
However, if you have complex journey tests that rely on a set of data or external services that are outside of your control then you may want to exclude these from your regression testing as having your tests fail on a regular basis does offset several the benefits.
Longer term benefits¶
Having a performance regression test pack does have longer reaching benefits outside of an application development lifecycle.
The benefits we have already discussed at length are primarily relevant during the application development lifecycle. What I mean by that is not only the initial development but also any further development that is not classed as business as usual.
Once an application is fully developed and running as a productions service and its functionality only increased in line with business change requests the testing of them is often sporadic and only accelerates if a defect is found in production.
By keeping all automated regression, not necessarily just performance regression, running on a regular basis ensures that should you need to rapidly accelerate your testing efforts then you have a working set of performance tests in a working environment all ready to go and the benefit and importance of this cannot be understated.
As your application matures in its role in production having a constant set of tests being regularly executed against the system can offer reassurance to the product owners and technology stakeholders that this level of constant awareness of the performance of the application will identify anything early that may impact it.
The importance of leaving a legacy cannot be overstated, sometimes QA teams devote a lot of effort to the performance testing, and it benefits the final product that performs well and meets all its non-functional requirements.
During the development phases the assets created are extensively used to ensure that performance is maintained and then the application is promoted to production, and the core QA team disappear off to other pieces of work.
If, or when, the QA effort needs to start again due to either additional business requirements or a regulatory change or because of a change to the way an external interface works, it can sometimes be difficult to determine what performance testing was run and the rationale behind it.
If the resources who originally ran the testing are no longer available, then having a working regression pack running at accurate testing volumes is a good start for any new resources who may be tasked with getting the performance testing running again.
Ownership¶
If we are going to leave a legacy from our performance testing in the form of a performance regression pack then this is going to need an owner.
The owner needs to be technically proficient to maintain the tests and needs to be aware of the plans for the application as well as being aware of any ongoing production concerns.
Theoretically if the regression testing is running outside of a development cycle then the maintenance effort is minimal as nothing in the application is changing so its more about being responsible for monitoring the results and ensuring that the response times are comparable with the previous ones.
Conclusion¶
Building a performance regression pack is a great idea, and the benefits it brings are numerous and even if you only benefit from one of them then the investment would have been worthwhile.
It’s a very simple thing to build especially when you have already developed the test scripts as part of your performance testing during application development and all you need to do is build a capability that will continue to run while the application is not being actively developed.
I guarantee that once you have built one and see the benefits that it can bring during development, after development, during business as usual and beyond it will ensure that a performance regression test is always something you develop as you move on to new application development.