Chaotic Performance Tests with JMeter
Building performance tests that conform to a very specific level of load and concurrency is a standard approach to performance testing.
You determine your peak levels of load and concurrency, and you build a test that meets this.
You build soak test and scalability tests that conform to pre-determined levels of load and concurrency, and you execute these alongside the other scenarios you build to meet your performance requirements.
This is the correct approach and conforms to most organisations approach to performance testing and something you should always do; but in addition its also a good idea to build performance tests that change on each execution and put your application under a random ever changing load profile as it is not uncommon for application usage to change when new features are added or if migrating to a new solution therefore randomising your performance testing can provide you useful information on how your application will react to conditions that do not conform to your standard approach.
This blog post will look at how you can make your JMeter tests more chaotic using a variety of samplers, timers and a beanshell server.
Chaos Testing¶
Chaos testing does not really have anything to do with this post except for its random and arbitrary nature.
To provide some context, Netflix development teams devised the concept of Chaos Testing and built Chaos Monkeys that would basically cause ‘chaos’ on the cloud-based infrastructure to test for resilience and availability failures.
Chaos testing is a common principle in DevOps structures and ensures that all infrastructure components are tested for availability regardless of any component failure.
The point I am trying to make is that the very prescriptive shutting down of infrastructure components and disaster recovery tests, whilst important, did not always uncover real world scenarios of failure.
Sometimes it is good to think of performance testing in this chaotic way in that tests that are random and test your application in a random way may uncover performance issues you would not find with the more common peak hour, scalability, and soak testing.
We are not suggesting for a minute that these formal tests are not required. Its just that combining these with random tests as well is something worth considering.
JMeter Controllers, Timers and Beanshell Server¶
So, to create chaotic performance tests in JMeter you must work with JMeters Controller, Timers and the Beanshell Server; this blog post will not spend time looking at all the various controllers and timers that JMeter has as this is not the point of the post, but it is worth reading these posts which will provide you with a good knowledge of how these JMeter components work:
Now to create chaos you don’t need to use all these elements of JMeter every time. You can use one, two or even all three if you want but individually, they can be adapted to run dynamic ever-changing tests.
Just like the tests themselves there is no specific, prescriptive way to do this, and we therefore cannot walk you through a specific way of accomplishing these chaotic tests what we can do is give you some examples of how we would approach it and give you enough information to start to look at these principles yourself.
Defining Chaos¶
The title may seem counter-intuitive as chaos by its definition is not defined but you are going to need some boundaries in the chaos.
For example, if you have an internal application, it may only be sized to support your internal user base which let’s say for example is 1,000 individual users, if your chaos simulates 10,000 user logons then it is chaotic but unrealistic and all it will do is cause your application to error.
Chaos should not cause application failure; it should stress your application in a random non-predictive way to try and understand the impact of unpredictable load, within system tolerances, on your application; there are other tests, scalability for example, that will look at stressing the system.
So, the first thing to do is define your limits and then let the chaos work within these boundaries, we have discussed what we would like to achieve now let’s build a test to demonstrate some examples of how you can build chaotic performance tests.
Building a test¶
When building a test for Chaotic Performance testing it is a good idea to start with one of your existing tests and then make some subtle changes to introduce randomness.
We will for the purposes of the post build ourselves a simple test using dummy samplers which will simulate an application being tested.
This test is purely made up and its sole purpose is to demonstrate chaos.
We have added several dummy samplers with a loop controller and an if statement.
At present the values that control the load are hardcoded with thread-group-one being:
The values in thread-group-two are:
The values in thread-group-three are:
The values in thread-group-four are:
The loop controller is set to:
The if statement is set to:
This is just a simple way of always evaluating to true and is there for the purposes of our simple test.
If we were to execute this test, we would always get the same outcome and if this was a genuine test that is not a bad thing as the objective of our performance testing is to place our system under test under representative production load.
Let’s look at the results.
We can see that our example load test generates 3000 samples across the test with the samplers running at a variety of rates.
And our load profile looks like:
One simple way to introduce a more chaotic test, and one worth mentioning, is to replace your hardcoded values with a random value which in JMeter is trivial.
Let’s now look at our Thread groups, Loop Controller and If Controller.
We can see that we have replace the hardcoded values with a random number whilst limiting the lower and upper boundaries.
We will not show all the thread groups, but we have changed them all in a similar fashion.
The Loop Controller follows a similar pattern:
The If Controller has a slightly more complicated solution where we firstly generate a random number between 0 and 10 and assign to a property as part of a JSR223 Sampler.
And then the If Controller uses a modulus expression to return true or false:
We will now run the test twice to show how we have created a more random, chaotic test albeit within an acceptable boundary range.
By increasing the lower and upper boundaries of the Random function you can increase the chaotic nature of your test.
That was a very simple way of introducing randomness so let’s explore others; before we move on, we are going to replace all our hardcoded values with properties so we can control them more easily and outside of the actual test itself.
Let’s now look at our Thread groups, Loop Controller and If Controller.
We can see that we have replace the random numbers with properties.
We will not show all the thread groups, but we have changed them all in a similar fashion.
The Loop Controller again follows a similar pattern:
The If Controller remains the same as it was before, but we change the JSR223 Sampler to use a property.
What we are effectively doing is allowing the random nature of our chaotic test be managed externally meaning we can create profiles that generate different kinds of chaos.
To demonstrate how we would execute our test now we need to build a flat file containing the values we want to replace our properties with; we could alternatively add these to the command line as we execute our test but as we have a significant number of properties the most elegant way to define them in a flat file.
This is an example of a file that would simulate what we have done to date in terms of volumes:
You can of course provide the same value for min and max to get a static value should you wish to do so and test at consistent volumes.
For the purposes of this post, we will start JMeter in UI mode to enable us to compare the results, you can of course include the -n switch to run the test in headless mode.
Using this command, where the script is called chaos-performance-testing.jmx and the properties file is called chaos-performance-test-properties.dat
./jmeter.bat -t chaos-performance-testing.jmx -p chaos-performance-test-properties.dat
We will now run the test to see that we can cause a chaotic performance test without hardcoded values, albeit in a random function, in our test.
We now can generate chaotic load profiles from a flat file, this then allows us to be more specific in how we target our load. If we look at our flat file again, this time with some annotation, we can see that we can target a specific part of the application with chaotic load while keeping the remainder of the load static.
This shows how we can create a much more inconsistent load on one part of the system whilst keeping the rest stable.
You can have multiple files all with different levels of chaos built in meaning that you can stress all or parts of you application under chaotic load.
These are only a few of the techniques that you could use to demonstrate how chaos can be achieved.
The links earlier in the post provide several techniques for controlling load all of which can be adopted to use random and flexible values to ensure that each execution differs albeit with parameters you control.
By using your existing tests to achieve this, and randomising the load, the effort to add this technique to your regular performance testing cycles is minimal and easy to manage any profile you wish to simulate.
Conclusion¶
As already stated in this post we are not suggesting that running your standard Peak Volume, Scalability, Soak Tests etc are not meaningful and beneficial; far from it.
What we are suggesting is that sometime you see production performance issues when the unexpected happens, something that you were not expecting and something that naturally you did not test for. By using this technique to create a chaotic and unpredictable load test you can maybe expose performance issues that you may have otherwise missed.
If the effort to create scenarios that may not happen was significant then the return against your investment would probably not make it worthwhile. But as your existing tests can be updated fairly simply using these techniques there is no reason to not at least experiment which chaos.
Contrary to what you may have heard chaos is a good thing especially when it comes to performance testing.
The JMeter project used in this blog post can be found here along with its properties file.