OctoPerf v15.1 - Continuous integration and new trend report
Version 15 of OctoPerf bring a huge focus on continuous integration and Devops practices. We've taken steps to highlight existing integrations, add many others and we've thrown an entirely new report type on top of it.
Continuous integration improvements¶
It's always been possible to automate OctoPerf tests using our existing plugins or simply through our API. That said the setup of those integrations comes with a learning curve. In order to make this curve as flat as possible we've added a new wizard and the possibility to get a PDF report automatically at the end of the test.
Easier integrations¶
On the runtime screen, the integrations panel at the bottom lists the main integrations you can use in order to execute your load tests outside of OctoPerf:
When you select one of them a very detailed wizard will popup and guide you through the entire setup. It covers everything from the installation of the third party tool to the setup of an automated OctoPerf test:
Those examples come with every workspace/project detail prefilled and also follow the best security guidelines regarding how to provide your API Key. In the example above, the actual API key is not in the XML file but will be provided in the command line along with guidelines on how to secure your API key or projets:
With this, we are confident that you will be able to execute your OctoPerf tests in the safest way possible while not having to spend large amounts of time learning an entirely new system.
Another example is for our HTTP API, here we will provide a command line that you can execute in order to run the test:
You can even choose the command/language you want to generate, that way you can get a working example in your context with a few clicks through our wizard.
And of course we'll improve those based on real uses cases and feedbacks, so make sure to get in touch with us if you encounter any challenge.
PDF report¶
It was already possible to generate the PDF report at the end of a test launched with our plugins or even with a test executed from the API.
In order to make this even easier we've added it to the notifications system as well. With that you are now able to ask OctoPerf to generate a PDF report and send it via email when the test is finished:
Combine this with our notification filters and you can get the PDF only for relevant tests, that way you avoid notification overload.
Trend report¶
OctoPerf's strong suit is already our reporting but that doesn't mean we can't improve it further. With this brand-new report type it is possible to compare up to 25 tests in a single report.
Comparison reports already give a lot of details but are limited to 4 tests, this new trend report takes a median approach with fewer details per test, but a larger number of tests. Our hope is that it will make more sense in the case of very regular tests for instance within a Devops or CI pipeline that includes load testing:
And of course every metric is compared to a Reference result of your choice:
And there's even new report items you can use to highlight differences:
You can select the Reference when you create the Trend report and then chose to have a static trend or one that auto-updates with every new test:
Data generation¶
When you want to run a test quickly with production-like data you often need to spend a lot of time creating meaningful datasets.
To simplify this process, you can now directly generate your fake datasets inside OctoPerf. This feature is available on any CSV variable:
First select your locale and the type of fake data you want.
Then you get an overview of the file that will be generated. Once finished, you can find it in the files menu and download/edit it to your needs.
UX changes¶
In this section are some of the minor changes we've introduced based on your feedbacks, so keep them coming!
Validation map¶
The validation panel inside the design page now comes with an interactive map, exactly like the location selector from the runtime screen:
With this change, new users will be able to better understand the purpose of the virtual user validation, which we hope will reduce errors down the line.
Time range visible in report summary¶
You can apply a time range filter to your test results, in order to see only a subset of the total duration of the test. The issue with this is that it's not obvious when such a filter is in place so things can get confusing quick.
To improve this, we've decided to display the time range on the test summary displayed on top of every report:
Now this graph shows 2 important details:
- The real end of the test versus the theoretical duration (in discontinued lines),
- The time range being displayed in all other graphs (with highlighted background colors).
Preview tab¶
When navigating through lists of items, a single click would select and a double click would open for edition. Our goal was to prevent too many tabs to open automatically when you click during your navigation because it could get confusing and/or slow. But a lot of the feedbacks were asking us to enable a single click to open any item.
We found that the best of the two worlds could be achieved by having a preview tab that opens on a single click, and updates when you click on another item. It's easy to recognize because it is always the first tab on display and it's named is italicized.
And if you want an item to remain, you can still double click to open it in another tab that stays available:
We've implemented this behavior for all lists of items inside OctoPerf.
Clickable elements more visible¶
Another issue we faced is that although a lot of the texts inside OctoPerf are interactable with a single click, it's not always clear until you've tried. To that end, we've underlined all interactable texts in order to suggest that those are hyperlinks (technically they're not) and as such can be clicked on.
In the screenshot below you can see that the user profiles are clickable but also the text under the button to Add first runtime property:
Full changelog¶
For the complete list of fixed bugs, please refer to OctoPerf 15.1 Release Notes.