Performance, Load and Stress Testing Explained
This is a typical questions QA Engineers and Devops may be asked during an interview. And we know how confusing these tests can be.
That's why we've written this article to explain once and for all the key differences between all types of performance testing.
As stated on Software Performance Testing, there are many different ways to test software including:
- Load Testing,
- Stress Testing (also called Torture Testing sometimes),
- Soak Testing: testing over long period of time,
- Spike Testing: simulate huge spike of load on a very short period of time,
- Scalability Testing: test how the applications scales horizontally or vertically.
Each type of testing has its own set of rules and goals. Let's explore which type of testing you should use, when and the results you get.
Let's start our visual tour!
Performance Testing¶
Introduction¶
All the type of testing mentioned above are in fact Performance Testing. It's the general category in which each sub-type of testing falls.
Typically, performance tests are conducted to validate:
- Software changes: that newly introduced changes into the tested software don't break it,
- Continuous Performance Validation: validate the performance of the system, not only correctness.
As Wikipedia explains it so well:
In software engineering, performance testing is in general, a testing practice performed to determine how a system performs in terms of responsiveness, scalability and stability under a particular workload.
Testing Methodology¶
This typically involves the following steps:
- Recording a simulation of the steps to perform on the target application,
- Running those simulation at large scale with thousands of concurrent users,
- Analyze Test Results to find bottlenecks, and other performance degradations.
I know how impatient you are so let's see them in greater details!
Load Testing¶
This is the simplest form of performance testing:
- Duration doesn't import much: Although a minimum duration is required for meaningfull results, the goal is mainly to reach a target load and check the system status,
- Rampup Load: usually, load is gradually increased until reaching a peak load which is maintained afterwards,
- Monitoring: server-side components are usually monitored during the load test to pinpoint any backend issue.
The goal of load tests is to validate that the system can handle a given load.
As it's general purpose, load testing is suitable for pretty much any simulation whose results should indicate how well the system behaves under a given load. Those tests are generally runs as part of a CI/CD cycle although in that context they might be simplified a bit for greater maintainability.
Soak Testing¶
This kind of test has the following characteristics:
- Long-running: involves testing a system with a typical production load, over a continuous availability period, to validate system behavior under production use,
- Production Load: simulate load which is as close to load seen in production as possible,
- May be shorter duration than in production: simulated load can be accelerated in order to test more quickly,
- Reach up to peak load during test: max simulated load should reach production peak loads, and not just average load.
This kind of test is also known as endurance testing. It involves testing the system for an extended period of time with load near to peak admissible loads. The system should maintain the performance over a long period of time.
The goal is to determine how the system behaves under sustained load.
Whenever you need to determine the absolute number of users a system can handle.
Soak Tests should be run regularly to check the system is maintaining a good endurance even after code updates.
Spike Testing¶
This kind of test has the following characteristics:
- Short-running: test is ran very quickly (usually a few minutes),
- Spike Load: load increases dramatically and very quickly to a huge surge.
The goal is to test the system resistance in the event of a sudden spike of concurrent users.
Spike Tests are suitable to prevent issues during short-term floods of users. Typically, this happens during events like the Black Friday on online shops. It can be during a TV ad campaign too, bringing a flood of users when the ad is run.
Stress Testing¶
Typically, stress tests involve:
- Always Increasing Load: load is increasing during the whole test until the servers being tested fall apart,
- Stepping load : it is recommended to increase load with steps so that you can maintain a stable level of load for some time before going to the next one. This will make analysis of server behavoir much easier for each step,
- Various Durations: test duration only depends on the rate of load increase you want to apply on the system,
- Goes well above expectations: load can increase well beyond what the system has been designed for.
The goal is to check robustness of a system in case of extreme load (greater than system capabilities). This may be followed by a light load test to check how much time the system needs to recover from the surge.
Recap Table¶
As you can see, the differences can be pretty much resumed in the following table:
| Test Type | Duration | Simulated Load | Usage |
|---|---|---|---|
| Load Testing | Variable | Slowly increasing, then maintain | As part of CI/CD cycles |
| Soak Testing | Long | Mostly maintain 90% peak load | Test system's resistance to sustained load |
| Spike Testing | Short | Surge increases with Spikes | Test how the system handles huge load on Ponctual Events |
| Stress Testing | Long | Always Increasing | Find system's max load; test system resilience |
The shape of the load curve really determines what kind of test you are running.
We hope this article will help you land the Performance Engineer job of your dreams!