Load Test Driven Development
We are going to explore whether Load Test Driven Development is an idea that would be worth pursuing for your organisation.
We will recap on what Test-Driven Development (TDD) is in the next section but fundamentally
Test-Driven Development is a philosophy and practice that involves building and executing tests before implementing the code or a component of a system
Now when you think about this, does it make sense to try and run a performance test before we have developed any code?
We think it does and we are going to explain why. For clarity we are not suggesting that Load test Driven Development should replace TDD but rather to compliment it.
Overview of Test-Driven Development¶
Let’s recap on what Test-Driven development is first so we can see how we can incorporate Load test Driven Development into this highly used agile framework.
Basically, TDD is a practise where you build and execute a test before developing any code or adding any component to your system under test.
This is part of the 'test-first' approach where building tests early allows you to build many fast automated tests and fewer slower manual tests, end-to-end, tests.
The steps in the process are defined below and have been taken from here:
- Write the test first, ensuring that the developer understands the required behaviour,
- Run the test and watch it fail. Because there’s no code yet, this may seem pointless, but it accomplishes two useful objectives: it verifies the test works, including any testing harnesses and demonstrates how the system will behave if the code is incorrect,
- Write the minimum amount of code needed to pass the test. If it fails, rework the code or the test until it passes routinely,
- Continue implementing new code until all tests pass. This step provides the developer with the confidence that their changes meet the current requirements and haven’t created an error in another part of the system,
- Refactor as necessary to ensure the design aligns with changing requirements (e.g., emergent design). Developers continually update their designs to ensure that evolving requirements and a growing codebase don’t lead to poor code quality.
How does Load Testing fit in¶
So, we now know what TDD is so how do we incorporate Load Testing?
We will explore the benefits in the next section but here we will focus on fundamentally how we write a performance test against something that does not exist, any tool specific reference will be for JMeter only.
The answer is you need to write very small, compartmentalised tests and there will be no opportunity to build any end-to-end flows, and this should not be expected.
It is worth noting that if you are expecting to simulate UI driven requests then this approach is not going to work for you, the benefits of Load test Driven Development only apply to the services that are built to support the UI interaction which are normally API’s or Database requests.
If the functionality being developed is an API :
- Build a HTTP Request that calls the API with the payload as per the design,
- Execute it and you get a 5xx error as the endpoint does not yet exist, this is what will happen if the test fails,
-
You want to capture this response so write a negative assertion to do so,
-
If you want to build a positive assertion for your responses, then you can use a Dummy Sampler to simulate the correct response against which you can check your response assertion is correct.
If the functionality is a Database request, then the principles are similar:
- Build a test that queries a database schema that does not exist ,
- Execute it and you get an error,
- Build a negative assertion for the error,
- Build a positive assertion based on a successful result.
The outcome is that you have developed a load test that can be executed during development. This brings many benefits that we will discuss below. The definition of a load test is that it generates concurrency as if it’s run as a single request then it does not differ from an automated functional test which if you are following a TDD approach you will have many of. Therefore, Load Test Driven Development needs to be run with an element of concurrency but there are limitations which we will discuss in the next section.
Clearly there are many variations on technology stacks, and these are examples of two; your ability to employ Load Test Driven Development depends very much on the technology being developed and the framework in which it is being developed.
Dev environments != production¶
If we want to incorporate a Load test Driven Development philosophy, then the tests will be built and executed in the development environment, and it is highly unlikely that the development environments will be sized anywhere near production or your pre-production environment and therefore this again challenges the benefits of Load test Driven Development.
The benefits do exist and sometime just by running a small number of concurrent users on an environment that clearly will not be able to support your production load can highlight issues with threading or code that has fundamental design issues where contention is seen. It can help the development teams to see concurrency through their code during development because mostly unit tests are single threads that exercise the functionality rather than the functionality with concurrency.
We are not talking about Load Test Driven Development replacing the need for formal performance testing we are advocating the benefits of writing tests before development.
Execution of tests¶
Now as we are advocating performance script development during code development the execution of these tests, once developed and the functionality written needs to be part of the code delivery pipelines.
Building tests as part of Load test Driven Development means that these tests need to be executed on each code commit to check for regression which is a fundamental to checking code quality through development.
We have some great posts about how you can incorporate JMeter tests into Jenkins Pipelines to support performance test execution in line with development. These can be found in the links below:
What are the benefits¶
We have looked at how we might approach Load Test Driven Development and basically concluded that we are writing tests before development and then routinely executing them during development to help with ensuring code quality.
We have also discussed the fact that these load tests need to be very specific and targeted to a particular piece of code rather than a journey test.
Let’s look at the other benefits Load Test Driven Development can bring to your organisation.
It helps you understand the application under test.¶
Being involved before code is developed will significantly improve your knowledge of the application being built. You are immersing yourself in the design and working alongside developers to get an understanding of how the application is working. This knowledge can help you as the product evolves to understand any performance issues later in the formal performance testing phases.
It also helps you establish performance as a guiding principle of the development and team mentality which will all lead to a better product.
Indexes¶
Database indexes play a huge part in application performance and whilst running tests before code is developed is not going to expose any missing ones, because they do not exist yet, thinking about them during development based on the queries being constructed is extremely useful and ensures that they are implemented as part of the design.
Exposing thread pool size issues¶
Another major factor in poorly performing applications is the way threads are managed and released by the code, effective management of thread pools ensure your application performs without the need to try and fix the problem by scaling up.
Not releasing connection is a major factor in memory consumption and normally is not found until you start to run your longer peak volume, soak test scenarios and this is much later in the project development cycles. Developing to manage thread pools effectively and not relying on out of the box configuration is something that can be addressed when using Load Test Driven Development.
Data growth to simulate production volumes¶
Not something obvious from early test creation but the reality is the longer your run your tests, those that create data will start to grow volumes in the database meaning that over time you will have more data in your application against which to test.
This is a very heavily caveated point as I am conscious that development environment databases are regularly truncated and undergo significant schema changes as the application evolves but the fact remains that the earlier you start to add data the greater chance you have of testing the indexes we discussed above.
Constantly looking for regression¶
As soon as the code exists to support your test and you have your first clean pass you are technically into a regression cycle where your test executes against all code changes.
You need to acknowledge that the design may change due to circumstances beyond anyone’s control and the code must be refactored. And so do your tests but nonetheless you are still performing regression against a piece of code. This means that performance is constantly monitored and checked and becomes part of the acceptance criteria for code promotion through the environments which is again instrumental throughout the development process. If performance issues are known when code is being developed, you save a lot of time on complex analysis conducted after the development completes.
Completely supports Agile Delivery¶
Agile delivery is clearly the ambition of many organisations as well as being the way many already operate and the inclusion of performance into this process is necessary.
There is little point developing and functionally testing the code if there are a set of performance tests at the end of development, or even at the end of each sprint because at that point the development effort has moved on and the team re-focussed.
Building your test suite¶
Ultimately whether you build tests early or later in the development cycles you are going to have to run a set of formal tests to satisfy your Non-Functional requirements and this is made a lot easier by using the simple focused performance tests you have been building through delivery.
Many of the tests can be used as they are and others may need to be altered but ultimately you are not starting your test script development from scratch, even if you do need to re-write several tests you will be doing so from a position of understanding; having been involved in writing the tests in line with development.
Conclusion¶
If your organisation works in a way that suits Test Driven Development, then you should be also including Load test Driven Development into your approach it may seem very early for performance testing but as we have tried to show in this post the involvement of someone who understands performance tuning being involved during development will undoubtedly improve the quality of it from a performance perspective.