Soak Test – A Practical Example
As performance testers Soak Testing is a test that is often overlooked.
We always think of Peak Volume tests and we always like to push the system to its limits in the form of a Scalability Test but often the Soak Test is forgotten about.
It’s a great test and it tells you many critical aspects about your application under load, things that no other test can and is critical in determining if you application is fit for production.
What is it?
It’s a test that run for a long period of time, what is a long period of time .. well it depends upon your application, organisation, production requirements, appetite for testing….
Soak Test Considerations¶
Lets consider the first 3; application, organisation and production requirements.
If you work in an organisation where the production systems are shut-down and re-started every night then your Soak Test has to last no longer than 24 hours.
If however your application is re-started weekly as part of a maintenance cycle then your Soak Test needs to last 7 days.
If you application is 24/7/365 then you are not going to test for a year, that is not practical.... My view is that 2 overnight cycles are required at the minimum so effectively a 3 day test.
Why do we test¶
- To expose memory leaks
- To check for suitable garbage collection policies
- To check that connection pools are being recycled
- To make sure log cycling is appropriate
- To check file handlers are being recycled
etc...etc...etc...the list goes on… really anything that will not be picked up on by tests that only last a couple of hours.
Building a test¶
Ok so we have discussed the reasons and benefits for this much undervalued test so lets think about how we test it and how we can leverage JMeter to support this.
Volumes¶
If you have built a Peak Volume Test and you understand your seasonal transaction and concurrency peaks then a Soak Test needs to be run at approximately 60% of these volumes to simulate volumes closer to your application averages.
JMeter Test Plan¶
We recently posted a Blog about how to use properties in JMeter to make Flexible and Configurable Test Plans, it is worth reading this and understanding the principles as we will use these principles to show how with the simple use of a JSR223 Sampler we can easily and elegantly turn your existing tests into a soak test that is easy and flexible to control.
Let’s start with our test, it is a simple tests consisting of 4 Dummy Samplers
You will notice that we have replaced the Thread Properties and Scheduler Configuration sections with Simplified Property Function, the rationale for this is discussed in our Blog Post on Flexible and Configurable Test Plans
The samplers have a Constant Throughput Timer to manage the transaction rate.
We are using Dummy Samplers so we don’t Soak Test a live web site for the purposes of this post
The Test Plan is not that important, and it could be any test that you need to meet your Non Functional Requirements
OK now to show you how easy it is to make this into a simple configurable Soak Test, firstly we need to add a JSR223 Timer to our test
This JSR223 Timer simply tells all threads in the Thread Group to sleep for a period of time, this needs to be a Simplified Property Function in order for the test to work and to help you understand how parameters can be abstracted from the test see our Blog Post Flexible and Configurable Test Plans
Initially we set the threadSleep Simplified Property Function in the JSR223 Timer above to be 0 as we do not want to pause the test and the dRation Simplified Property Function that controls the Test Duration to an arbitrary value.
These will both be overwritten by code as the test executes and we will discuss this next
To demonstrate our parameter values the properties file we will use looks like this:
We now have a simple Test Plan configured to run for 300 seconds at a rate of 600 transactions per minute, this does not seem like much of a Soak Test!
Now to turn this into a Soak Test we simply add a separate Thread Group and a JSR223 Sampler... simple really, let’s do this now.
You will notice that this Thread Group contains a single Thread and uses the same Simplified Property Function value as the dRation Simplified Property Function in the octo-perf-soak Thread Group this ensures that both finish at the same time.
Now we add our JSR223 Sampler that does all the calculations.
We will look at our script in a moment, but it is important to understand what we are trying to achieve.
We are going to simulate a application that is available between 08:00 and 23:00 and we want to run our test across 2 overnight cycles as per the recommendation earlier in this post
To do this we will
- Start our Soak Test at some point on Day 1
- Require that it executes until 23:00
- Re-start at 08:00 the next day where it executes until 23:00
- Pause until 08:00 the following day
- Completing at 23:00 on the third day
Let’s visualise the test¶
Whilst simple, this is a good representation of a Soak Test, let’s look at the script that manages the execution.
We will look at it in an organised way, but don’t worry we’ll publish the fully annotated code at the end
Firstly two helper methods to help us in our calculations, one to calculate seconds to a particular time on the current day
public long currentDayMilliCount(int dayOfMonth, int hourOfDay) {
Calendar c = Calendar.getInstance();
c.add(Calendar.DAY_OF_MONTH, dayOfMonth);
c.set(Calendar.HOUR_OF_DAY, hourOfDay);
c.set(Calendar.MINUTE, 0);
c.set(Calendar.SECOND, 0);
c.set(Calendar.MILLISECOND, 0);
long howMany = (c.getTimeInMillis()-System.currentTimeMillis());
return howMany;
}
And the second to return the number of milliseconds for any number of hours duration
public long calculateMilliCount(int noHours) {
return noHours * 3600 * 1000;
}
OK that’s the helper methods build, now let’s build the main method that will manage the execution flow, this is broken into small chunks to allow us to discuss in detail.
public void manageSoakTestExecution(int noDaysExecution, int startTime, int dailyDuration) {
We create the method with three parameters
- no of days we want to execute the test for
- time we want to start daily
- duration of a daily test.
To start with we return the number of milliseconds until 23:00 on the first day of the test using one of our helper methods, remember the test could start at any time so day ones calculations are slightly different from the remaining days
long milliToTestStopDayOne = currentDayMilliCount(0, 23);
We will show all these calculation times in our visualised test
We also need to determine how long until midnight on day one, so we can calculate the sleep time for the test on the first day
long milliToMidnightDayOne = currentDayMilliCount(1, 0);
We calculate the sleep time for our test from midnight each day until the desired start time
long milliStartOfDay = calculateMilliCount(startTime);
We calculate the execution time for each full day of the test
long milliOnDayExecutionTime = calculateMilliCount(dailyDuration);
We calculate the remaining sleep time for each full day of the test
long milliEndOfDay = calculateMilliCount(24 - (startTime + dailyDuration));
We calculate the duration of the whole day element of the test
long milliAllDaysDuration = calculateMilliCount(24 * noDaysExecution);
Now we have our basic calculations we can manage the test, we firstly set the duration of the Soak Test by changing the dRation parameter. This is effectively the time to midnight on day one of the test plus the number of days of solid execution we wish to run for.
We do this by overwriting the dRation Simplified Property Function that manages the execution time
long soakTestDuration = milliToMidnightDayOne + milliAllDaysDuration;
props.put("dRation", soakTestDuration.toString());
We now start our test with two Thread Groups running with the manage-test-duration Thread Group controlling the octo-perf-soak Thread Group
We now pause our manage-test-duration Thread Group, until it is required to make changes to Simplified Property Function in the octo-perf-soak Thread Group, the next action will be to pause the test when we have been running for the time set in milliToTestStopDayOne
We therefore tell the manage-test-duration sampler to sleep until needed
Thread.sleep(milliToTestStopDayOne);
We iterate our noDaysExeuction variable here using a for loop and follow the same execution pattern for each subsequent day.
for(i=1; i<=noDaysExecution; i++) {
Once we have waited for the end of the test on Day 1 we need to pause our octo-perf-soak Thread Group until it needs to starts again which is effectively the remaining sleep time for the test for a day plus the sleep time for the next day when the test starts again
We overwrite the threadSleep variable in the manage-sleep-time JSR223 Timer
props.put("threadSleep", milliEndOfDay + milliStartOfDay);
And we pause our manage-test-duration Thread Group until we need to perform another action, this is effectively the time when the next day’s octo-perf-soak Thread Group iteration needs to stop
Thread.sleep(milliEndOfDay + milliStartOfDay + milliOnDayExecutionTime);
}
}
And that is it!!!!
Summary¶
All we are doing is using one Thread Group to control another using Simplified Property Function values and thread sleep time
All that remains to be done is to call the method, with appropriate parameters to manage the Soak Test
manageSoakTestExecution(2, 8, 15);
Full JSR223 Sampler extract¶
Full JSR223 Sampler extract:
/* Function to calculate seconds to a particular time on the current day */
public long currentDayMilliCount(int dayOfMonth, int hourOfDay) {
/* Return the number of milliseconds from the current time to a particular hour of the day */
Calendar c = Calendar.getInstance();
c.add(Calendar.DAY_OF_MONTH, dayOfMonth);
c.set(Calendar.HOUR_OF_DAY, hourOfDay);
c.set(Calendar.MINUTE, 0);
c.set(Calendar.SECOND, 0);
c.set(Calendar.MILLISECOND, 0);
long howMany = (c.getTimeInMillis()-System.currentTimeMillis());
return howMany;
}
public long calculateMilliCount(int noHours) {
/* Return the number of milliseconds for any number of hours duration */
return noHours * 3600 * 1000;
}
public void manageSoakTestExecution(int noDaysExecution, int startTime, int dailyDuration) {
/* We will build an execution schedule for the Soak Test which will manage the test */
/* Firstly we need to determine how long the test needs to run for on the current day */
/* The test could be started at a any time of the day and in our example we want it to */
/* stop before midnight and not start again before a certain time on the following day */
/* Using our functions we calculate how much time between now and our desired end time */
long milliToTestStopDayOne = currentDayMilliCount(0, 23);
/* We also need to determine how long until midnight so we can calculate the sleep time */
/* for the test on the first day, we use the same function with alternative parameters */
long milliToMidnightDayOne = currentDayMilliCount(1, 0);
/* We calculate the sleep time for our test from midnight each day until the desired start time */
long milliStartOfDay = calculateMilliCount(startTime);
/* We calculate the execution time for the test */
long milliOnDayExecutionTime = calculateMilliCount(dailyDuration);
/* We calculate the remaining sleep time for the test */
long milliEndOfDay = calculateMilliCount(24 - (startTime + dailyDuration));
/* We calculate the duration of the whole day element of the test */
long milliAllDaysDuration = calculateMilliCount(24 * noDaysExecution);
/* Now we have our basic calculations we can manage the test */
/* We firstly set the duration of the soak test by changing the dRation parameter */
/* The first thing we do is set the test duration for the whole test day */
/* This is effectively the time to midnight on the start day of the test plus the number of days */
/* of solid execution we wish to run for */
/* We do this by overwriting the dRation property that manages the execution time */
long soakTestDuration = milliToMidnightDayOne + milliAllDaysDuration;
props.put("dRation", soakTestDuration.toString());
/* We now pause our manage-test-duration sampler, until it is required to change further parameters*/
/* The next action will be to pause the test when we have executed for the time set in the milliToTestStopDayOne variable */
Thread.sleep(milliToTestStopDayOne);
/* We iterate our noDaysExeuction variable here and follow the same pattern for each day */
for(i=1; i<=noDaysExecution; i++) {
/* Once we have waited for the end of the test we need to pause our */
/* octo-perf-soak sampler until it starts again which */
/* is effectively the remaining sleep time for the test for a day plus the sleep time for the next day when */
/* the test starts again*/
/* We overwrite the threadSleep variable in the manage-sleep-time sampler */
props.put("threadSleep", milliEndOfDay + milliStartOfDay);
/* And we pause our manage-test-duration sampler until we need to perform another action */
/* This is effectively the time when the next days tests needs to stop */
Thread.sleep(milliEndOfDay + milliStartOfDay + milliOnDayExecutionTime);
}
}
/* Call the function to manage the soak test */
manageSoakTestExecution(2, 8, 15);