Playwright vs JMeter
In our quest to provide a comprehensive suite of tools for our load tester community, we, at OctoPerf, developed a full integration with Playwright. Playwright is an end-to-end testing library that has the ability to automate real browser actions. Which means that you will be able to do exactly that on OctoPerf, on top of all the advanced performance monitoring and report features that it already has.
In this article we will start with introducing browser based testing to better understand the advantages it can provide to a load tester. We will then compare a load testing campaign made with browser based and one with protocol based Virtual Users (VUs) to fully understand the differences.
Browser Based testing¶
What is browser based testing ?¶
Automated Browser based testing has been around for many years. Functional testers have been using that process to detect potential problems on their applications, since it gives the ability to simulate a virtual user launching a browser and performing actions on the interface. The automation of that process saves a lot of time and money, and allows for a faster and reliable QA, which in turn accelerates the application delivery process.
With all of that in mind, it's easy to understand the appeal of that technology for load testers.
Why browser based load testing ?¶
To understand why browser based testing might be used, we first need to talk about protocol based testing.
Protocol based VUs are, overwhelmingly, the main type of scripts that are used in the load testing community. In OctoPerf, we use JMeter to generate protocol based VUs. They consist in the capture of every requests generated by the browser while performing actions on an application, which are then executed to generate load. It has a lot advantages and provides great insight into the performances of your application under stress, but it could be criticized for one reason: Those users are not starting an actual browser. That fact means that the tester will have to handle himself some actions that were automatic on a browser, such as correlating tokens, fields ids etc.
Contrary to their protocol counterpart, browser-based VUs actually launch browser instances to perform the recorded user journey. In other words, the user loads the interface through the browser and uses it instead of manually sending HTTP requests.
Starting a browser means that we will get responses time for the application server side AND the user client side which, in theory, is more realistic. Especially since more and more actions are handled client-side in modern applications as frameworks like Angular are dominating the market right now. It also means that the tester won't have to handle dynamic parameters, which could save time in the scripting/maintenance phase, especially on some frameworks where they are generated through local JavaScript.
That being said, let's put all those assumptions to the test by comparing the two technologies on the exact same load testing campaign.
Browser Based vs Protocol testing¶
The campaign will be performed on Realworld an angular demo web application. Angular was chosen because it's prone to heavy client-side scripts executions. We will simply access it's homepage then login to the website. If you are new on Playwright, here is a comprehensive starter guide we published on our blog. It will require some setup to install on your local machine.
Recording¶
JMeter Virtual User¶
Recording a protocol VU can be achieved through HAR files, which can be generated through a browser. Every requests generated during the user journey are recorded and can be imported to OctoPerf. Ending up with requests instead of actions can make the result feel less clear, but the possibility offered by OctoPerf to create and name containers during the recording fixes it.
Playwright¶
Recording on playwright can be done using specific external recorders. Actions made on the browser will be recorded (clicking on buttons, typing in text fields), which feels intuitive since it's what a real user will do.
Parametrization and correlations¶
JMeter Virtual User¶
Parametrization can be done through OctoPerf's graphical user interface, thanks to the variables menu. CSV files can be uploaded, variables can be generated and directly used in requests, which make it easy to maintain and share with other testers as the parameters are centralized and common to every script in the project.
As already explained, the tester will have to handle parameters handling and generation made by the browser, so some additional work will usually be required. OctoPerf helps with this process with extractors that are enhanced with helpers, such as automatic REGEXP generators.
On top of that, correlation rules can be created which automates and greatly facilitate this process. In the rare situations when javascript is generating dynamic values that are absolutely necessary to the application, we will have to go through the execution of code which is possible thanks to JSR223 actions.
Playwright¶
Parametrization on Playwright can be done through external tools, we have a whole article in our blog on the subject. It will be done through code which requires some technical knowledge. Consequently, those customizations will also be more difficult to maintain and explain to other team members if necessary.
Since Playwright is browser based, dynamic parameters such as tokens or javascript generated values will be handled automatically, which can be a huge time saver.
Results¶
The load generated will be exactly the same for both tests. It will be 10 minutes long, 5 concurrent VUs with a 1 minute pacing time.
For both technologies, the results will be accessible through OctoPerf's analysis interface.
JMeter Virtual User¶
Metrics will be available for every action in your Virtual User, including requests, which allows for a deep understanding of the way your application is handling the load on the server-side.
Playwright¶
Initially, metrics will be available for your Playwright actions and the entire execution. If more details are needed you will have to go through the setup of aggregators, which will allow for more informations on individual items such as resources download. You'll also be able to get more context through the use of the Playwright Trace Viewer.
Results comparison¶
We can see the response time differs between the two types of VUs.
The total user journey execution is way longer with the browser based VU, which can be explained by the browser being actually launched, as explained earlier, but also because this time also includes the Playwright's script command execution time, adding a small overhead. Looking at each of the 2 steps, the main time difference can be noticed for the homepage loading, since that's when all the javascript is executed.
But there is also a major difference in terms of resources consumptions between the 2 tests. Playwright VUs are almost saturating the load generator's (LG) CPU while the JMeter based VUs are barely using it:
In fact, after multiple tests, we observed that a load generator that could handle 1000 protocol VUs could only handle approximately 5 Playwright Vus.
We are finally understanding why browser-based testing, while being available for a very long time, never really took-off.
Over the years, browsers have progressively demanded higher amounts of system resources, such as CPU and memory usage, on our machines. This heightened resource consumption is primarily due to the increasing complexity of web applications. Because of this, browser based testing requires more and more machines to generate the targeted load, which drives the costs up.
Another side-effect is that the more powerful your machine, the faster your browser will execute, which is also true for the load generators. In other words, you will get different results depending your LGs configurations, which goes against the argument that this technology is more realistic.
Conclusion¶
We now understand the strengths and weaknesses of both types of load testing. Browser based load testing might seem convenient at first, but its heavy resource consumptions is in the end difficult to justify for heavy loads, not only in terms of costs, but also environmentally. The modern trend is to try to reduce company's carbon footprint, which doesn't fit with load tests starting hundreds of machines for relatively small loads. On top of that, in the EU, the Corporate Sustainability Reporting Directive (CSRD) now requires a large number of companies to publish reports regarding their environmental impact. Protocol testing is most likely here to stay.
But instead of trying to pit them against each other, one way to go would be to use both of them in a way that would combine their benefits. The bulk of the load can be generated by the protocol VUs, while a small amount of browser based VUs can be executed at the same time, measuring the response time under load including everything happening client-side. That way, you get all the benefits of both sides without the inconveniences.