JMeter vs SoapUI

There are many functional and load testing tools available on the market. That's great!

The most renewed open-source tools are certainly JMeter and SoapUI.

But, I'm sure you agree to say that it's difficult to know which one best suits your needs:

• What features has JMeter?
• What are the pros of using SoapUI?
• Which tool has best community? User experience? Script maintainability?
• Should I use JMeter or SoapUI? Or maybe both?

Gain significant insight on JMeter and SoapUI differences by reading this blog post that compares them on many different fields:

• Tools Presentation: to know more about both these open-source QA applications,
• Project Resources: which tool is maintained more actively and has the largest community,
• Features Comparison: which of the two has the best features,
• Setup: how each tool can be installed,
• Getting Started Script: compares how to design scripts with both tools,
• and even more!

Ready for some action? Let's go!

Elevate your Load Testing!

Request a Demo

Tools Presentation

You may not be familiar with both tools or simply be new to the world of functional and performance testing solutions. This chapter briefly describes each open-source solution, starting with Apache JMeter...

JMeter Introduction

JMeter is developed by the open source Apache Software Foundation. It can be used as a functional and performance testing tool for a variety of services, with a focus on web applications.

Its primary goal is to simulate a heavy load on servers. But JMeter can also be used as a unit-test tool and for some functional testing as well.

It's a full featured test IDE that perfectly supports most popular web protocols:

• including HTTP/HTTPS,
• SOAP,
• REST,
• FTP,
• LDAP,
• and more.

JMeter architecture is based on plugins, allowing developers to easily extend it to suit their needs. It allows for fast test scenario recording (from Web Browsers and native applications), parameterizing and debugging.

JMeter has been first released in 1998, so this project is 20 years old! To offer enterprise-grade features this open-source project is used by several companies like CA BlazeMeter and of course OctoPerf.

SoapUI Introduction

SoapUI is an open-source desktop application first released in 2005.

It is focused on API Quality Assurance and used for functional and load testing of REST and SOAP-based Web Services over HTTP.

Features include:

• Functional API testing,
• Web Service simulation,
• Security QA,
• and Load Testing.

But don't be fooled by its name, SoapUI not only supports SOAP but also REST, JMS, AMF, as well as make any HTTP(S) and JDBC calls.

Its basic version is a free open source application. But a commercial edition that offers automation and productivity improvements, called SoapUI NG Pro, is also available. Both version are primarily developed by SmartBear Software.

Should I use JMeter or SoapUI?

Both are well established open-source applications offering a large panel of features and answering a wide range of QA problematics. Also, they are both 10+ years old and backed-up by companies. Thus ensuring that commercial services are available and that they will be maintained during the coming years.

Note: This article focuses on free versions/usage of both tools.

JMeter being more focused on load testing and SoapUI on functional testing, I think that both tools are worthy of your time.

But don't take my word for it and keep on reading to have an in-depth knowledge of their advantages and drawbacks!

Project Resources

This section checks which project is maintained more actively, has the most active community and the best documentation.

JMeter Commits, Documentation and Community

As I said JMeter is an open-source project, released under the Apache 2.0 license. It has 15K commits made by 39 contributors. And as we can see on OpenHub, it's codebase has steadily grown since its creation to reach almost 500K lines today:

The JMeter documentation is really great. Each feature of the tool is explained and all possible settings are detailed.

Moreover, many blogs about JMeter are available:

• OctoPerf's JMeter blog,
• BlazeMeter's blog,
• Ubik's Blog (FR) by the main contributor of JMeter,
• and another contributor's blog: Milamber (FR).

JMeter also has a large community on StackOverflow with 10K questions asked, and a Google Group (FR).

All you need to learn and even master JMeter - or get help when you get stuck!

SoapUI Commits, Documentation and Community

SoapUI's open-source project is released under the European Union Public Licence.

It has 7K commits made by 82 contributors. And as we can see on OpenHub, it's codebase is stable since 2012 with 250K lines of code:

The SoapUI documentation is also well made. It includes several tutorials that are great to get started.

You may find SoapUI's blog handy during your API testing journey as well as Ole Lensmar's blog (creator of SoapUI). And the StackOverflow community is here to answer your questions (3K so far).

Support Winner

You are more likely to find support from the community using JMeter as it is an older project used in many commercial products.

SoapUI still have a large community with plentiful resources to help you learn it.

Features Comparison

JMeter Features

JMeter supports a wide variety of applications/server/protocol types:

• HTTP/HTTPS Web applications, whatever the server programming language (Java, NodeJS, PHP, ASP.NET, ...),
• SOAP/REST Web Services,
• JDBC database connections,
• File Transfer Protocol,
• Message-oriented middlewares by using Java Message Service,
• Mail (SMTP, POP3 or IMAP)
• And even generic TCP connections and OS native processes!

JMeter is a Java desktop application. Thus it's completely portable and can be installed on any OS. It can also be run in command-line mode (Non GUI) to launch automated tests thanks to its numerous integrations like Maven and Jenkins. JMeter generates HTML reports.

As you can see JMeter already has a lot to offer. And in case that is not enough or you, it's based on plugins. So you'll most likely always find a way to make it suit your needs, even if that means writing your own plugin.

For example at OctoPerf we wrote a plugin to generate a custom user load. There are also many other free plugins available at JMeter Plugins as well as commercial ones for complex protocols (Vaadin, GWT, Flex, ...) made by Ubik.

SoapUI Features

SoapUI also supports many protocols and application types:

• SOAP/REST Web Services over HTTP/HTTPS,
• JDBC database connections,
• JMS,
• AMF (Adobes ActionScript Messaging Format used by Flash/Flex applications),
• and OAuth 1 & 2 authentications.

Just like JMeter, SoapUI is a Java desktop application installable on any OS. It can also be run in command-line mode to execute automated tests using, for example, Maven and Jenkins.

You can import Postman Collections directly into SoapUI and export test results in several formats (HTML, PDF, DOC, ...).

SoapUI also features assertions and response extractors. Several plugins are available, most of them being maintained by SmartBear.

Features Comparison Table

The following Table compares the features supported by both tools.

	JMeter	SoapUI
HTTP/HTTPS	Yes	Yes
WebSocket	Yes*	Yes
Rest Apis	Yes	Yes (Discovery)
Soap Services	Yes	Yes (WSDL)
FTP	Yes	No
JDBC	Yes	Yes
JMS	Yes	Yes
TCP	Yes	No
AMF (Flex)	Yes*	Yes
Vaadin/GWT	Yes*	No
HTTP Recorder	Yes	Yes
Functional Tests	Yes	Yes
Load Tests	Yes	Yes
GUI Scripting	Yes	Yes
Command-Line Mode	Yes	Yes
Assertions	Yes	Yes
Response Extractors	Yes	Yes
Plugins	Yes	Yes
Reporting	HTML	HTML, PDF, DOC, ...

Note

* external plugins or services may be required (potentially commercial ones).

Features Verdict

And the winner is... JMeter! JMeter has a few more features than SoapUI and a better protocol support.

Setup

JMeter and SoapUI tools both require a Java JRE installed on the machine.

JMeter Setup

• Install Java on your machine if not already present,
• Download JMeter Archive,
• Unzip the archive,
• Run ./bin/jmeter.sh executable. (.bat on Windows)

SoapUI Setup

• Download SoapUI,
• Execute the installer.

Setup Verdict

SoapUI's comes in several packages:

• An installer that includes Java,
• A tarball like JMeter's.

So when it comes to Tool Setup, SoapUI wins!

Getting Started Script

Let's see how to easy it is to create a simple test in both JMeter and SoapUI.

Our script will make successive calls to HttpBin:

1. Get the current date from https://now.httpbin.org/,
2. Extract the date from the JSON response,
3. Send a GET request to https://httpbin.org/get with the current date in query parameters,
4. Assert that the response contains the date.

It's a dummy but dynamic script, used to evaluate how convenient are JMeter and SoapUI.

JMeter Scripting

A JMeter project consists in a tree of Test Elements whose root is a Test Plan.

To compose your test:

1. Simply right click on a tree node (left panel of JMeter),
2. Select Add in the contextual menu,
3. Select the category,
4. And finally select the Test Element.

This will add a new Test Element to your project. There are several categories of Test Elements in JMeter, all detailed in the documentation:

• Samplers used to send request of various kinds (HTTP, JDBC, JMS, etc.),
• Logic Controllers like If conditions, containers and transactions,
• Post Processors, mostly variable extractors and scripts,
• Assertions like JSON or XPATH assertions,
• Timers to slow down the test execution,
• Listeners to debug the test and view results.

TL;DR, the completed JMeter project is available here: HTTPBin.jmx. You can download the JMX file and open it in JMeter while reading this section.

So to start designing our simple script, we will add several Test Elements to our Test Plan:

• Thread Group: from the Add > Threads (Users) contextual menu when right clicking on the Test Plan,
  • HTTP Request: Contextual menu on the Thread Group > Add > Samplers > HTTP Request,
    • JSON Extractor: Add > Post Processors > JSON Extractor,
  • Another HTTP Request: Add > Samplers > HTTP Request,
    • JSON Assertion: Add > Assertions > JSON Assertion,
  • View Result Tree: Add > Listeners > View Result Tree.

The created Test Plan should look like this:

Configuring the Test Elements

The first Test Element we added is a Thread Group. It holds the configuration for the load test with the number of concurrent users and iterations.

This is the most basic Thread Group, several others let you create custom load policies with concurrent users ramp up and down.

Keep the default configuration with only one concurrent user executed one time. This is perfect for debugging the script.

Now skip to the first HTTP request. Configure it to make a call to https://now.httpbin.org/ by filling in the Protocol and Server Name:

Click on the JSON Extractor and set its JSON path to $['now']['slang_date']. Also set the name of the created variable to date:

This will inject the extracted value in the variable date. In the next elements of this test, the date variable can be injected using the ${date} syntax.

Let's configure the second HTTP request, this time by setting the server protocol and name but also the request path /get and the query parameter date.

Finally configure the JSON Assertion to check the presence of the date in the server response by using th JSON path $['args']['date']:

Don't forget to click on Additionally assert value and to set today in the Expected Value field.

Debugging the Load Test

Time to test things out! Click on the last test element, the View Result Tree and click on the green play button (or press CTRL + R).

The View Result Tree lets you check the request/response for each sampler in your Test Plan. As you can see here everything went fine: Both requests were executed (you can check the server response by clicking on them), and the assertion was OK.

Now that we are sure our script is properly configured, it's time to run a load test!

Running the Load Test

Running a load test is pretty straightforward: simply edit the Thread Group to add more concurrent users and iterations, then click on the Play button once more.

Note: our test plan does not contain any timer, you should consider adding them when conducting a real load test!

To analyse your load test, you might want to add two Listeners: the Summary Report and the Graph Results.

The Summary report displays useful information like:

• Minimum, maximum and average response times,
• Sent/received bytes and rate,
• Number of request and errors,
• The standard deviation,
• etc.

If you set different names for your samplers, you will be able to differentiate them in the summary table:

Another handy Test Element to analyse test results is the Graph Results:

And there are many others, just check the Add > Listeners contextual menu to try them.

SoapUI Scripting

To get started easily with SoapUI I recommend to import one of the embedded tutorials and to have a look at the documentation tips.

TL;DR, the completed SoapUI project is available here: HttpBin-soapui-project.xml. You can import this project in SoapUI while reading this chapter.

Creating the Project

Once SoapUI is started, the first thing to do is to create a project:

You can have multiple projects opened at the same time.

To rename the created project, simply select it and press F2 or right-click on it and select the Rename option.

Note: Most actions in SoapUI are made using the right-click contextual menu.

Creating the REST services

In SoapUI, before making successive requests you first need to create REST services.

Right-click on your project and select the menu New REST Service from URI. A popup appears:

Do this twice to create REST Services for both https://now.httpbin.org/ and https://httpbin.org/get.

Both services appears under the HttpBin project. You can open a request by double clicking on it. Then press the green launch button to have a look at the server response. The response body is visible in the right panel, under the tab JSON:

Let's add the date query parameter to the https://httpbin.org/get service:

1. In the project tree, double click on the last node of the https://httpbin.org/get service,
2. The editor is displayed,
3. Click on the green plus button and select date for the name and some_value for the value,
4. Send the request, you should see the query parameter returned in the JSON response.

Creating the Test Suite

Now your REST services can be used in the real scripts: Test Suites / Test Cases.

To create a Test Case:

1. Right click on the project,
2. Select New Test Suite,
3. Rick click on the created Test Suite,
4. Select New Test Case.

You should now have the following project tree:

The sequence of HTTP calls must be added in the Test Steps section. This can be done simply by dragging them from the REST services and dropping them in the Test Steps.

Then we will use a Property Transfer to extract the date from the first request and inject it in the second one. Add it using the right-click > Add Step > Property Transfer on the Test Case 1 and configure it like that ($['now']['slang_date']):

You can click on the launch button to test it. And don't forget to set the Path Language to JSON Path like I did for my first attempt!

We also need to add an assertion on the last request to ensure that HttpBin sent us the date back. Double click on the request (under the test case), the editor appears, click on the green plus button to add an assertion. Select JSON Path Match and configure it to match the date $['args']['date']:

Running the Test Case

It's time to run our Test Case. Double-click on it the in the project tree and click on the green launch button:

As you can see in the screenshot above (TestCase Logs section), everything went fine:

1. The first request is sent to now.httpbin.org,
2. The date parameter is transferred,
3. The second request is sent to httpbin.org/get,
4. The assertion validates the presence of the date in the server response.

Running a Load Test

That's nice, but this blog is mostly about load testing! Let's see how we can run a load test using SoapUI.

In the project tree, under the Test Case, right-click on Load Test and select New LoadTest in the contextual menu. The default configuration starts 5 concurrent users for 60 seconds, with a 1 second delay between each execution. You can run it by pressing the green launch button:

By default SoapUI displays several information:

• Minimum, maximum and average response times,
• Sent bytes and rate,
• Number of request and errors,
• etc.

Also you can click on the Statistics chart icon to display a graph of the results:

Scripting Winner

That's probably a matter of personal taste, but I feel more comfortable scripting using JMeter.

I'm not fond of SoapUI's many windows on the right panel: it hinders readability. Also having to first create REST services then add them to test cases is a bit cumbersome in SoapUI.

On the other hand I like being able to execute each request individually and check the server response immediately: In JMeter you have to debug the whole script.

Conclusion

Both JMeter and SoapUI are great open-source tools! I personally prefer JMeter for its extensive features and community.

But you might prefer SoapUI in case you do more functional tests of Web Services (especially SOAP ones), JMeter being primarily oriented toward load testing of Web Services and applications.

But don't trust me on this. Test them both and make your own opinion!

Want to become a super load tester?

Request a Demo