OctoPerf v12.8 - Datadog, Json Path and sub samples
It's been a while since the last update post in july 2021, not that we haven't updated OctoPerf since then but the additions we've made are not easy to share in a blog. Allow me to take an example.
JMeter import¶
The thing that kept us the most busy over the years is finding the perfect way to import a JMeter project into our data model. We need to do this in order to:
- Allow you to manipulate your virtual users in our interface even when they come from JMeter,
- Execute each Threadgroup/Virtual user in a separate docker container to make our runtime resilient and scalable (this allows us to predict resource consumption better and allocate machines accordingly),
- Offer a configurable report with filters instead of a static HTML.
But we must make sure to maintain the same behavior for all functionalities. And at the same time we must avoid negative impacts on our non-JMeter users (like added UI complexity, or regressions). That seems simple enough at first glance but it gets harder when you consider that JMeter allows you to put any configuration anywhere but with a different scope. For instance you can have header managers configured this way:
Once imported in OctoPerf each Threadgroup will be a distinct Virtual User, and we need to consider carefully what to do with these headers, we must find a way to preserve the same behavior in OctoPerf.
To be more specific, we need to copy the top one into each Virtual user or request because otherwise if several JMX were imported in the same project with Header managers in the test plan, there could be conflicts once we regenerate the JMX. (ie several header managers in the test plan that could have the same header, but different values.) The one inside the Threadgroup, we need to keep it there as a Generic action but the one under the request, we will use it to generate the headers displayed on the request in our UI.
Finding the proper way to handle all the situations like this requires some trial and error and what I've presented here is the result of several iterations. And this is just for a single JMeter config element, so as you can imagine this can get out of hand quickly.
Which is why it is important for us to maintain:
- An open channel through the chat, to identify such issues quickly and understand the missing content/behavior,
- Unit tests to cover all the situations encountered in the past in order to avoid regressions when we need to make a change later (we have just reached 10k Unit tests recently).
With this we are able to fix any of these issues in a matter of hours and make OctoPerf more and more resilient. So if you check the changelog at the end of this post you'll see we've dealt with a lot of these recently (especially around encoding and fragments) because more and more companies trust OctoPerf with their load tests. And we've kept the same reactivity as always when it comes to fixing them.
What's new¶
We've been working on stabilizing the current version of OctoPerf as much as we can to prepare for our next big step. We will soon tell you more about this.
But meanwhile we've also added a new assertion, a fully integrated Datadog external reporting, better support for streaming plugins and many other minor features.
Improvements¶
Datadog external reporting¶
The Datadog JMeter plugin is now fully supported in OctoPerf and available from our UI:
This allows you to send all the load test metrics into Datadog and OctoPerf at the same time. This allows different teams to use the tool of their choice to analyze the metrics. Or you could simply use it to aggregate everything in a single tool.
Json Path assertion¶
Up until now only the Response assertion was natively available in OctoPerf, we've now also added the Json path assertion and of course it comes with a Json selector similar to the Json extractor:
We'll probably add other extractors in the future, let us know in the chat which ones you would like to see the most.
Sub samples analysis¶
In JMeter everything that has a result is either a sampler or a controller, but in general results are labeled as samples. Sample results are already collected by OctoPerf as part of each test, but some JMeter mechanisms or plugins use sub samples.
Sub samples are either children of the main request, like the automatic resources attached to their parent. In other situations, the main samples doesn't even have a result, instead it creates a set of sub samples, like in the case of Streaming samplers:
OctoPerf used to discard these sub samples for performance reasons but as you can see in the above screenshot, they are now fully integrated in our reporting.
Test Executor role¶
Our workspaces allow you to define access rights on projects and other resources. But the roles used to be only reader/writer which is sometimes not flexible enough, especially when different persons maintains the scripts and run the tests.
This is why we've made it more flexible by creating a new Executor role that:
- Cannot edit anything in the Design,
- Can create Runtime scenarios,
- Can execute tests,
- Can configure test reports and templates,
Http Keep-alive¶
JMeter offers a keep-alive option on HTTP requests. This option was not available in OctoPerf because we made sure to:
- Enable keep-alive on all requests in order to reuse the TCP session,
- Disable keep alive on the last request of the virtual user in order to trigger a new TCP session during the next iteration.
This way we were able to mimic the behavior of a different browser on every iteration of the virtual user. But in some situation keep-alive is either not relevant or must always be activated, which is why we've made this setting more flexible:
We offer 3 modes:
- Auto is the one described above,
- Always off,
- Always on.
We do not recommend Always on since it doesn't represent what real users would do.
GET with Body¶
In our UI request body were always POST/PUT request bodies, but some applications actually use GET requests with bodies. That is not exactly against the specifications but it's also not supposed to happen which is why it wasn't allowed in OctoPerf.
It is now allowed under the Post Data tab, and by default we set it to none in order to preserve the current behavior of other features:
More info on how to manipulate request bodies can be found in our documentation.
Notifications improvements¶
The Notification screen allows you to send a message when specific events occur. We added a lot of information to this message over the years, making it difficult to read lately. In order to fix that we've improved the layout for slack and teams:
The test name (here Stress test) is a click-able link to the test report in OctoPerf.
Bench report variables¶
When creating Report templates, it is now possible to use variables to add some dynamic content. Those will be updated based on the context of the result on which the template is applied:
${PROJECT_NAME}${PROJECT_DESCRIPTION}${REPORT_NAME}${REPORT_DESCRIPTION}
Non HTTP error details¶
The error details of plugins and other actions will now be available in the test report.
Full changelog¶
For the complete list of fixed bugs, please refer to OctoPerf 12.8 Release Notes.