Which environment to use?
We can all agree that running a realistic load test is important to assess server performance correctly. One of the aspects of such a test is the testing environment. It must be selected carefully based on your requirements.
The sooner the better¶
Since I began my career as a tester I've been told it is better to test early to prevent costly defects. But sadly I must admit I ran a lot more tests in production than in an early Q&A environment. This usually happens because load testing is disregarded and only conducted at the last moment. However, testing early comes with a lot benefits even if it is also quite different than a typical load test.
Component testing¶
Early performance tests are often referred as component testing. These tests are conducted when the development is not even finished. At this stage you can only test the different components one by one to assess if their performance under load meets your expectations.
Testing early allows you to fix any issue for a lower cost because the application is still under development. This is the best way to avoid stacking performance issues to the point where they would be difficult to diagnose. It allows your developers to fix the performance issues before they switch to another project or during the warranty period in case of a 3rd party developer. Plus by checking performance on a regular basis during development, you will also be able to size your production architecture better.
The drawbacks of component testing are mainly the lack of big picture :
- No complete page response time.
- No way to assess global server side behavior under load.
- Mutualized components behavior is also unknown.
That's why you still need to run a complete load test if you want to make sure your application performs well.
Production versus Pre-production¶
Running tests in an environment close to production will bring a lot of insight on how your application performs. Of course before running such tests, you need to consider if the environment you plan on using would be realistic. You should consider :
- Is the environment comparable to production ? (same number of machines, same configurations, etc...)
- Is the database populated enough, with realistic data ?
- Is this environment properly maintained like production ? Are there the same background tasks that might hamper performance ?
- Is anything mutualized with production ? (firewall, proxies, network, etc...)
Looking at things this way, it might seem safe to assume that the closest to production the better, but pre-production testing also has it's benefits :
- Easier preparation : DataSet creation and/or data creating user profiles are no longer a problem. Database restorations should also be easier to conduct.
- Easier test schedule : You can run your tests almost any time provided no other team uses the environment. This give a lot more flexibility if the test campaign lasts longer than expected.
- Lower risk of side effects : You are less likely to overload a mutualized component critical to another production application. In case of virtual machines, you usually end up on a different physical machine, thus limiting the risks of side effects.
Testing in production¶
In my early days as a tester, I recall struggling to explain to an architect why the word "testing" quickly followed by the word "production" came out of my mouth. He had some valid points about data creation, statistics and such, but as no realistic pre-production environment was available, production was the best option. This is always something to consider, because coming up with a testing strategy in a production environment can be difficult. You might have to:
- Schedule the tests far outside business hours or during the week end
- Put the application under maintenance
- Ask the DBA team to backup the database before the tests
- Synchronize with the security team to avoid DDOS protections working against your load generators
- Synchronize with the infrastructure team to have someone available to reboot the servers in case something goes wrong.
You must also be careful of what your test might do to other applications. In virtualized environments for instance, the physical machines can be shared amongst several applications.
Of course the obvious case when you can use the production to test is when the the production environment exists but is not yet opened to the public. It happens when applications are first deployed or when migrating the infrastructure. Sadly such situations are quite rare, but the result you can get by doing so are very accurate.
So what then ?¶
Test early if you can afford it, the results will always be better. But never forget that testing early does not replace a complete load test in a realistic environment.
In the end, the most complicated decision remains:
- Do I risk an unrealistic load test in pre-production ?
OR
- Do I risk overloading my production environment ?