Cloud testing vs Internal testing
The rise of cloud platforms lead to a major evolution for load and performance testing. Since generating load requires a powerful infrastructure for a short period of time, one might even argue this is the perfect use case. But since we started OctoPerf we had a lot of feedback from users wanting to test from their own machines for a lot of good reasons.
Web application does not mean web-site¶
First of all, not all web applications are available on the internet. In particular test environments, which are usually mostly meant to be used internally. As testing the application before its deployment is important, the load tests are often done on such environments.
Also, large companies have a wide range of intranet applications that can't be accessed from outside their network. Of course they need tests and in that case testing from the cloud does not make much sense.
Internal testing¶
When the application is not public at all, cloud testing should be ruled of. But even for public applications it is important to test internally as well. When facing a bottleneck you can't identify, a common testing protocol would be to eliminate as many layers of complexity as possible for the next test. Then you add them again one by one and see which one causes the issue. This is why running the first test from an internal machine is a good policy. That way you bypass network issues and work on other topics. When you assessed that the application is performing well then it is time to test from the cloud.
Cloud testing¶
Of course Cloud testing through a SaaS solution offers many benefits:
Realistic¶
Testing from the cloud allows you to test your application in conditions very close to what your users can expect. You can generate load from different places in the world at the same time. That way you can see how your application behaves, how your cache is propagated by your CDN, etc...
Less side effects¶
There is less risk that the test infrastructure hampers the performance of the application under test or any other production application. When testing internally there is always a risk that your test machines overload the network or any resource that could be used by another machine. This is particularly true if you are close to the servers, for instance if your test machines are virtualized on the same physical hosts.
High volumes¶
Cloud testing also allows you to generate much more load than internally. A proper cloud testing tool is scalable and does not requires you to setup large amounts of machines. This means that you get more flexibility and at the same time avoid internal costs and delays related to machines.
Enhancement and updates¶
When you install a load testing architecture internally, it can be tedious to migrate it to a newer version. Also when you do the upgrade you bear the risk of making a mistake which could prevent you from running tests. A cloud tool is always kept up to date without you knowing it and the maintenance period is minimum. Plus you always benefit from the newest features as soon as they are available.
Contractual flexibility¶
Most on premises load testing softwares will require you to buy a perpetual license. As the cost of such a license is usually quite high, it will require a long negotiation. A SaaS tool offers more flexibility by working with smaller amounts, on demand tests, subscriptions.
Cloud testing and cloud providers¶
Still you should be careful when testing an application that is also hosted in the cloud. For instance testing from Amazon EC2 to Amazon EC2 will leads to shorter response times than what you can expect. Because internal communication between datacenters is usually privileged. This is why you should always choose a different cloud provider than the one hosting the application.
To sum up¶
Test first as close to your servers as you possibly can, but if your application is public, running a test from the cloud is a must.
This is why in OctoPerf we provide both on premises and cloud testing.