JMeter response assertion
When running a protocol based load test with JMeter, it is always a challenge to make sure the server responses are correct. The default validation is only done on HTTP response codes based on their value:
OK < 400 <= KO
And since 4XX and 5XX HTTP response codes are error messages it makes sense. But it's easy to understand why this is not enough.
What to expect with no assertions¶
Take any web application with a login, when you enter invalid credentials you might get something like this:
But looking at the HTTP response we can see it's actually a 200 code:
So based on the default validation this will be considered ok and the script will continue even if you've configured it to stop on error. And this will lead to errors at some point because you're not logged in, and they will be more difficult to understand since they occur later on.
Well, assertions allow you to fail this step so that you quickly understand where the problem comes from. In this situation we could check that we do not have an invalid login message using this assertion:
And as a result we can see this:
Which makes it a lot easier to debug, since we now know exactly why it failed.
How to configure Assertions¶
Most of this section is also covered by JMeter's documentation on assertions, feel free to have a look at it.
Apply to¶
This is where you can define the scope of this assertion. Main sample only is the most common setting, this way the assertion will be applied only to the response to samplers not to sub samples. Sub samples in case of HTTP requests mainly refers to embedded resources. JMeter variable is useful to assert if a post processor contains the right value.
Response field to test¶
This completes the scope by allowing you to only assert on a portion of the request/response. The most common setting is Text response since it refers to the body of the response. Ignore status is also important when you expect an invalid response code (4XX or 5XX). For instance some authentication mechanism will give you a 403 then redirect you to a login page.
Pattern matching rules¶
Substring should be used when looking for a text message.
The others allow the use of regexp inside the pattern, just be careful of ORO characters like . * ?.
The Not checkbox allows for a negative search, in case you don't want to have a particular error message. If you use several patterns, they will all need to be present for the assertion to succeed, you can use the Or checkbox if you want only one out of them to be enough.
Patterns to test¶
This is where you enter the text or regexp to look for. Here is an example:
Best practices¶
Avoid confusing text¶
Avoid a negative search for a short text like "error". There might be a lot of occurrences of this word in any given page, for example in a script embedded within the page. And this will not accurately indicate a problem.
Instead, try to look for a more specific error message:
Internal Server Error
\d\d\d Error
Search for functional OK/KO messages¶
A good way to make sure you have a valid response is to identify a functional message from the application that will not be there in case something is wrong. For instance on any Webstore, after a successful purchase you will get a "Thank you" message. This clearly indicates that the whole purchasing process went well.
Another good way is to identify the application error messages and validate that you do not get one. Whenever you run into an issue when navigating through the application, try to add this to the list of assertions on this page.
Search for dynamic values¶
Whenever your script involves a login step, the username is usually displayed on the next page. That's a very good way to make sure the login worked fine and that the right user is connected. If you are using a CSV dataset it's easy to add another column with the username and search for it after the login. In case you have not selected substring in the Pattern matching rules, you can escape ORO chars that might be part of the name this way:
${__escapeOroRegexpChars(${username})}
Do not overdo it¶
Although it is tempting to assert every server response, assertions are heavy on CPU for your load generators. Try to validate meaningful responses, for instance limit it to one per transaction/page of the application. Also as we've seen earlier, it might be enough to ensure that the final step went well since it usually means that all prior actions are also OK. In the end you have to balance the amount of effort you're willing to put into it, the test infrastructure you have and the relevance of the assertions.
Combine it with Action to be taken after a sampler error¶
The main goals of assertions is to identify errors as soon as possible to avoid overwhelming your servers with invalid requests. To achieve this, I strongly recommend to configure your thread group to at least Start next thread loop on error:
This way, if you placed your assertions well, every failed thread will log only one error instead of continuing and getting many of them. It makes the analysis much easier and also helps with understanding the issue.
Assertions in OctoPerf¶
As you've seen assertions are very powerful but they can be difficult to configure. In OctoPerf we provide a simple way to define them based on your selection in the response content. For instance if you select some text to validate, it will be automatically checked at runtime:
And when going to the advanced tab you get the same possibilities than in JMeter. Plus the assertion will automatically be pre-filled and special ORO chars will be escaped for you:
Also a click on check will let you know if this assertion will work on the recorded or validation content.
Then at runtime every assertion that fails will log a message in the error table where you can analyze the details:
And you can even render the response to find out quickly what's the problem:
As usual we want to bring the most value on top of what you can already do in JMeter. And since assertions are already quite powerful, we focused on their ease of use. Feel free to have a look and tweet us your feedback!