--- title: JMeter vs SoapUI url: https://blog.octoperf.com/jmeter-vs-soapui/ date: '2018-06-05' authors: - gpereira categories: - Load-Testing tags: - SoapUI - JMeter - Open Source - Quality Assurance - SOAP - REST - BlazeMeter - FTP description: Which open source tool between Apache JMeter and SoapUI best suits your needs? Learn the key differences between those tools and make your choice. --- # JMeter vs SoapUI There are many **functional and load testing tools** available on the market. That's great! [The most renewed open-source tools](https://blog.octoperf.com/ultimate-guide-23-free-load-testing-tools-reviewed/) are certainly **JMeter and SoapUI**. But, I'm sure you agree to say that it's difficult to know **which one best suits your needs**: - What features has [JMeter](https://jmeter.apache.org/)? - What are the pros of using [SoapUI](https://github.com/SmartBear/soapui)? - Which tool has best community? User experience? Script maintainability? - Should I use JMeter or SoapUI? Or maybe both? **Gain significant insight on JMeter and SoapUI differences** by reading this blog post that compares them on many different fields: - [Tools Presentation](#tools-presentation): to know more about both these open-source QA applications, - [Project Resources](#project-resources): which tool is maintained more actively and has the largest community, - [Features Comparison](#features-comparison): which of the two has the best features, - [Setup](#setup): how each tool can be installed, - [Getting Started Script](#getting-started-script): compares how to design scripts with both tools, - and even more! Ready for some action? Let's go! ## Tools Presentation You may not be familiar with both tools or simply be new to **the world of functional and performance testing solutions**. This chapter briefly describes each open-source solution, starting with Apache JMeter... ### JMeter Introduction ![Apache JMeter](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/apache-jmeter.png "Apache JMeter") JMeter is developed by the open source Apache Software Foundation. It can be used as a functional and performance testing tool for a variety of services, with a focus on web applications. **Its primary goal is to simulate a heavy load on servers.** But JMeter can also be used as a unit-test tool and for some functional testing as well. It's a full featured test IDE that perfectly supports most popular web protocols: * including HTTP/HTTPS, * SOAP, * REST, * FTP, * LDAP, * and more. JMeter architecture is based on plugins, allowing developers to easily extend it to suit their needs. It allows for fast test scenario recording (from Web Browsers and native applications), parameterizing and debugging. JMeter has been first released in [1998](https://en.wikipedia.org/wiki/Apache_JMeter#Releases), **so this project is 20 years old!** To offer enterprise-grade features this open-source project is used by several companies like CA BlazeMeter and of course OctoPerf. ### SoapUI Introduction ![SoapUI by SmartBear](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soap-ui-by-smartbear.png "SoapUI by SmartBear") [SoapUI](https://en.wikipedia.org/wiki/SoapUI) is an open-source desktop application first released in 2005. It is focused on API Quality Assurance and **used for functional and load testing of REST and SOAP-based Web Services over HTTP**. Features include: * Functional API testing, * Web Service simulation, * Security QA, * and Load Testing. But don't be fooled by its name, SoapUI not only supports SOAP but also REST, JMS, AMF, as well as make any HTTP(S) and JDBC calls. Its basic version is a **free open source application**. But a commercial edition that offers automation and productivity improvements, called SoapUI NG Pro, is also available. Both version are primarily developed by SmartBear Software. ### Should I use JMeter or SoapUI? **Both are well established open-source applications offering a large panel of features and answering a wide range of QA problematics.** Also, they are both 10+ years old and backed-up by companies. Thus ensuring that commercial services are available and that they will be maintained during the coming years. > **Note:** This article focuses on free versions/usage of both tools. JMeter being more focused on load testing and SoapUI on functional testing, **I think that both tools are worthy of your time**. But don't take my word for it and keep on reading to have an in-depth knowledge of their advantages and drawbacks! ## Project Resources This section checks which project is maintained more actively, has the most active community and the best documentation. ### JMeter Commits, Documentation and Community As I said JMeter is an open-source project, released under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0). It has 15K commits made by 39 contributors. And as we can see on [OpenHub](https://www.openhub.net/p/jmeter), it's codebase has steadily grown since its creation to reach almost 500K lines today: ![JMeter code base](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-codebase.png "JMeter codebase") The [JMeter documentation](https://jmeter.apache.org/usermanual/) is really great. Each feature of the tool is explained and all possible settings are detailed. Moreover, many blogs about JMeter are available: * [OctoPerf's JMeter blog](https://blog.octoperf.com/categories/jmeter/), * BlazeMeter's blog, * [Ubik's Blog](https://www.ubik-ingenierie.com/blog/author/pmouawad/) (FR) by the main contributor of JMeter, * and another contributor's blog: [Milamber](https://blog.milamberspace.net/index.php/jmeter-pages) (FR). JMeter also has a large community on [StackOverflow](https://stackoverflow.com/questions/tagged/jmeter) with 10K questions asked, and a [Google Group](https://groups.google.com/forum/#!forum/jmeter-fr) (FR). All you need to learn and even master JMeter - or get help when you get stuck! ### SoapUI Commits, Documentation and Community SoapUI's open-source project is released under the [European Union Public Licence](https://en.wikipedia.org/wiki/European_Union_Public_Licence). It has 7K commits made by 82 contributors. And as we can see on [OpenHub](https://www.openhub.net/p/soapui), it's codebase is stable since 2012 with 250K lines of code: ![SoapUI code base](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-codebase.png "SoapUI codebase") The [SoapUI documentation](https://www.soapui.org/docs/getting-started/) is also well made. It includes several [tutorials](https://www.soapui.org/docs/rest-testing/) that are great to get started. You may find [SoapUI's blog](https://blog.smartbear.com/category/soapui/) handy during your API testing journey as well as [Ole Lensmar's blog (creator of SoapUI)](https://olensmar.blogspot.com/). And the [StackOverflow community](https://stackoverflow.com/questions/tagged/soapui) is here to answer your questions (3K so far). ### Support Winner **You are more likely to find support from the community using JMeter** as it is an older project used in many commercial products. SoapUI still have a large community with plentiful resources to help you learn it. ## Features Comparison ### JMeter Features **JMeter supports a wide variety of applications/server/protocol types:** * HTTP/HTTPS Web applications, whatever the server programming language (Java, NodeJS, PHP, ASP.NET, ...), * SOAP/REST Web Services, * JDBC database connections, * File Transfer Protocol, * Message-oriented middlewares by using [Java Message Service](https://en.wikipedia.org/wiki/Java_Message_Service), * Mail (SMTP, POP3 or IMAP) * And even generic TCP connections and OS native processes! JMeter is a Java desktop application. Thus it's completely portable and can be installed on any OS. It can also be run in command-line mode (Non GUI) to **launch automated tests thanks to its numerous integrations** like [Maven](https://github.com/jmeter-maven-plugin/jmeter-maven-plugin) and [Jenkins](https://wiki.jenkins.io/display/JENKINS/How+to+run+JMeter+with+Jenkins). JMeter generates HTML reports. As you can see JMeter already has a lot to offer. And in case that is not enough or you, it's based on plugins. **So you'll most likely always find a way to make it suit your needs, even if that means writing your own plugin.** For example at OctoPerf we wrote a plugin to generate a [custom user load](https://github.com/OctoPerf/thread-group). There are also many other free plugins available at [JMeter Plugins](https://jmeter-plugins.org/) as well as commercial ones for complex protocols (Vaadin, GWT, Flex, ...) made by [Ubik](https://ubikloadpack.com/). ### SoapUI Features **SoapUI also supports many protocols and application types:** * SOAP/REST Web Services over HTTP/HTTPS, * JDBC database connections, * JMS, * AMF (Adobes ActionScript Messaging Format used by Flash/Flex applications), * and OAuth 1 & 2 authentications. Just like JMeter, SoapUI is a Java desktop application installable on any OS. It can also be run in command-line mode to execute automated tests using, for example, Maven and Jenkins. You can import Postman Collections directly into SoapUI and export test results in several formats (HTML, PDF, DOC, ...). SoapUI also features assertions and response extractors. Several plugins are available, most of them being maintained by [SmartBear](https://github.com/SmartBear). ### Features Comparison Table The following Table compares the features supported by both tools. | | JMeter | SoapUI | |:--------------|:------:|:---------------:| | HTTP/HTTPS | Yes | Yes | | WebSocket | Yes* | Yes | | Rest Apis | Yes | Yes (Discovery) | | Soap Services | Yes | Yes (WSDL) | | FTP | Yes | No | | JDBC | Yes | Yes | | JMS | Yes | Yes | | TCP | Yes | No | | AMF (Flex) | Yes* | Yes | | Vaadin/GWT | Yes* | No | | HTTP Recorder | Yes | Yes | | Functional Tests | Yes | Yes | | Load Tests | Yes | Yes | | GUI Scripting | Yes | Yes | | Command-Line Mode | Yes | Yes | | Assertions | Yes | Yes | | Response Extractors | Yes | Yes | | Plugins | Yes | Yes | | Reporting | HTML | HTML, PDF, DOC, ... | !!! note \* external plugins or services may be required (potentially commercial ones). ### Features Verdict And the winner is... **JMeter**! JMeter has a few more features than SoapUI and a better protocol support. ## Setup JMeter and SoapUI tools both require a [Java JRE](https://openjdk.org/install/) installed on the machine. ### JMeter Setup - [Install Java](https://openjdk.org/install/) on your machine if not already present, - Download [JMeter Archive](https://jmeter.apache.org/download_jmeter.cgi), - Unzip the archive, - Run `./bin/jmeter.sh` executable. (.bat on Windows) ### SoapUI Setup - Download [SoapUI](https://www.soapui.org/downloads/latest-release/), - Execute the installer. ### Setup Verdict SoapUI's comes in several packages: - An installer that includes Java, - A tarball like JMeter's. So when it comes to **Tool Setup, SoapUI wins**! ## Getting Started Script Let's see how to easy it is to **create a simple test in both JMeter and SoapUI**. Our script will make successive calls to [HttpBin](https://httpbin.io/): 1. Get the current date from `https://now.httpbin.io/`, 2. Extract the date from the JSON response, 3. Send a GET request to [https://httpbin.io/get](https://httpbin.io/get) with the current date in query parameters, 3. Assert that the response contains the date. It's a dummy but dynamic script, used to **evaluate how convenient are JMeter and SoapUI**. ### JMeter Scripting **A JMeter project consists in a tree of Test Elements whose root is a Test Plan.** To compose your test: 1. Simply right click on a tree node (left panel of JMeter), 2. Select _Add_ in the contextual menu, 3. Select the category, 4. And finally select the Test Element. This will add a new Test Element to your project. There are several categories of Test Elements in JMeter, all detailed in the [documentation](https://jmeter.apache.org/usermanual/component_reference.html): * **Samplers** used to send request of various kinds (HTTP, JDBC, JMS, etc.), * **Logic Controllers** like If conditions, containers and transactions, * **Post Processors**, mostly variable extractors and scripts, * **Assertions** like JSON or XPATH assertions, * **Timers** to slow down the test execution, * **Listeners** to debug the test and view results. > **TL;DR, the completed JMeter project is available here: [HTTPBin.jmx](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/HTTPBin.jmx).** You can download the JMX file and open it in JMeter while reading this section. So to start designing our simple script, we will add several Test Elements to our Test Plan: * Thread Group: from the _Add > Threads (Users)_ contextual menu when right clicking on the Test Plan, * HTTP Request: Contextual menu on the _Thread Group > Add > Samplers > HTTP Request_, * JSON Extractor: _Add > Post Processors > JSON Extractor_, * Another HTTP Request: _Add > Samplers > HTTP Request_, * JSON Assertion: _Add > Assertions > JSON Assertion_, * View Result Tree: _Add > Listeners > View Result Tree_. The created Test Plan should look like this: ![JMeter Test Plan](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-test-plan.png "JMeter Test Plan") #### Configuring the Test Elements The first Test Element we added is a Thread Group. It holds the configuration for the load test with the number of concurrent users and iterations. > This is the most basic Thread Group, several others let you create custom load policies with concurrent users ramp up and down. ![JMeter Thread Group](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-default-thread-group.png "JMeter Thread Group") Keep the default configuration with only one concurrent user executed one time. This is perfect for debugging the script. Now skip to the first HTTP request. Configure it to make a call to `https://now.httpbin.org/` by filling in the _Protocol_ and _Server Name_: ![JMeter HTTP request now](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-now-request.png "JMeter HTTP request now") Click on the JSON Extractor and set its JSON path to `$['now']['slang_date']`. Also set the name of the created variable to `date`: ![JMeter JSON Post Processor](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-json-post-processor.png "JMeter JSON Post Processor") **This will inject the extracted value in the variable _date_. In the next elements of this test, the _date_ variable can be injected using the `${date}` syntax.** Let's configure the second HTTP request, this time by setting the server protocol and name but also the request path `/get` and the query parameter `date`. ![JMeter HTTP request get](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-get-request.png "JMeter HTTP request get") Finally configure the JSON Assertion to check the presence of the _date_ in the server response by using th JSON path `$['args']['date']`: ![JMeter JSON Assertion](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-json-assertion.png "JMeter JSON Assertion") Don't forget to click on _Additionally assert value_ and to set `today` in the _Expected Value_ field. #### Debugging the Load Test Time to test things out! Click on the last test element, the View Result Tree and click on the green _play_ button (or press CTRL + R). ![JMeter View Result Tree](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-view-result-tree.png "JMeter View Result Tree") **The View Result Tree lets you check the request/response for each sampler in your Test Plan.** As you can see here everything went fine: Both requests were executed (you can check the server response by clicking on them), and the assertion was OK. Now that we are sure our script is properly configured, it's time to run a load test! #### Running the Load Test Running a load test is pretty straightforward: simply edit the Thread Group to add more concurrent users and iterations, then click on the Play button once more. > **Note**: our test plan does not contain any timer, you should consider adding them when conducting a real load test! **To analyse your load test, you might want to add two Listeners: the Summary Report and the Graph Results.** The Summary report displays useful information like: * Minimum, maximum and average response times, * Sent/received bytes and rate, * Number of request and errors, * The standard deviation, * etc. If you set different names for your samplers, you will be able to differentiate them in the summary table: ![JMeter Summary Report](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-summary-report.png "JMeter Summary Report") Another handy Test Element to analyse test results is the Graph Results: ![JMeter Graph Results](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/jmeter-graph-results.png "JMeter Graph Results") And there are many others, just check the _Add > Listeners_ contextual menu to try them. ### SoapUI Scripting To get started easily with SoapUI I recommend to import one of the embedded tutorials and to have a look at [the documentation tips](https://www.soapui.org/docs/getting-started/10-tips-for-the-soapui-beginner/). > **TL;DR, the completed SoapUI project is available here: [HttpBin-soapui-project.xml](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/HttpBin-soapui-project.xml).** You can import this project in SoapUI while reading this chapter. #### Creating the Project **Once SoapUI is started, the first thing to do is to create a project:** ![SoapUI Create project](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-create-project.png "SoapUI Create Project") You can have multiple projects opened at the same time. To rename the created project, simply select it and press _F2_ or right-click on it and select the _Rename_ option. > **Note:** Most actions in SoapUI are made using the right-click contextual menu. #### Creating the REST services **In SoapUI, before making successive requests you first need to create REST services.** Right-click on your project and select the menu _New REST Service from URI_. A popup appears: ![SoapUI REST services](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-create-rest-service.png "SoapUI REST services") Do this twice to create REST Services for both `https://now.httpbin.io/` and `https://httpbin.io/get`. Both services appears under the _HttpBin_ project. You can open a request by double-clicking on it. Then press the green _launch_ button to have a look at the server response. The response body is visible in the right panel, under the tab `JSON`: ![SoapUI Launch REST services](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-rest-service-launch.png "SoapUI Launch REST services") Let's add the `date` query parameter to the `https://httpbin.io/get` service: 1. In the project tree, double click on the last node of the `https://httpbin.io/get` service, 2. The editor is displayed, 3. Click on the green _plus_ button and select `date` for the name and `some_value` for the value, 4. Send the request, you should see the query parameter returned in the JSON response. ![SoapUI Add Query Parameter](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-add-query-parameter.png "SoapUI Add Query Parameter") #### Creating the Test Suite **Now your REST services can be used in the real scripts: Test Suites / Test Cases.** To create a Test Case: 1. Right click on the project, 2. Select _New Test Suite_, 3. Rick click on the created Test Suite, 4. Select _New Test Case_. You should now have the following project tree: ![SoapUI Create Test Suite](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-create-test-suite.png "SoapUI Create Test Suite") The sequence of HTTP calls must be added in the _Test Steps_ section. This can be done simply by dragging them from the REST services and dropping them in the _Test Steps_. **Then we will use a _Property Transfer_ to extract the date from the first request and inject it in the second one.** Add it using the right-click > Add Step > Property Transfer on the _Test Case 1_ and configure it like that (`$['now']['slang_date']`): ![SoapUI Property Transfer](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-property-transfer.png "SoapUI Property Transfer") > You can click on the _launch_ button to test it. And don't forget to set the _Path Language_ to JSON Path like I did for my first attempt! We also need to **add an assertion on the last request to ensure that HttpBin sent us the `date` back**. Double click on the request (under the test case), the editor appears, click on the green _plus_ button to add an assertion. Select _JSON Path Match_ and configure it to match the date `$['args']['date']`: ![SoapUI Configure Assertion](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-configure-assertion.png "SoapUI Configure Assertion") #### Running the Test Case It's time to **run our Test Case**. Double-click on it the in the project tree and click on the green _launch_ button: ![SoapUI Run Test Case](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-run-test-case.png "SoapUI Run Test Case") As you can see in the screenshot above (TestCase Logs section), everything went fine: 1. The first request is sent to `now.httpbin.org`, 2. The _date_ parameter is transferred, 3. The second request is sent to `httpbin.org/get`, 4. The assertion validates the presence of the _date_ in the server response. #### Running a Load Test That's nice, but this blog is mostly about load testing! Let's see **how we can run a load test using SoapUI**. In the project tree, under the Test Case, right-click on Load Test and select _New LoadTest_ in the contextual menu. The default configuration starts 5 concurrent users for 60 seconds, with a 1 second delay between each execution. You can run it by pressing the green _launch_ button: ![SoapUI Run Load Test](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-run-load-test.png "SoapUI Run Load Test") By default SoapUI displays several information: * Minimum, maximum and average response times, * Sent bytes and rate, * Number of request and errors, * etc. Also you can click on the _Statistics_ chart icon to display a graph of the results: ![SoapUI Load Test Statistics](https://blog.octoperf.com/img/blog/jmeter-vs-soapui/soapui-load-test-statistics.png "SoapUI Load Test Statistics") ### Scripting Winner That's probably a matter of personal taste, but **I feel more comfortable scripting using JMeter**. I'm not fond of SoapUI's many windows on the right panel: it hinders readability. Also having to first create REST services then add them to test cases is a bit cumbersome in SoapUI. On the other hand I like being able to execute each request individually and check the server response immediately: In JMeter you have to debug the whole script. ## Conclusion **Both JMeter and SoapUI are great open-source tools!** I personally prefer JMeter for its extensive features and community. But you might prefer SoapUI in case you do more functional tests of Web Services (especially SOAP ones), JMeter being primarily oriented toward load testing of Web Services and applications. But don't trust me on this. **Test them both and make your own opinion!**