An Amazing Year
2016 was an exciting year for OctoPerf! We’ve really pushed the platform to an all new level thanks to your amazing feedback. It's been over a year and a half now working towards a more accessible performance testing solution built for literally everyone. Being a new player in this market can be a real challenge, but it’s also quite rewarding when you receive so much positive attention. Since so much has transpired in 2016, we would like to take a moment to summarize all the cool features we’ve added this past year:
First, we must remember that when 2016 began, OctoPerf only executed tests from the cloud. But we had to address both types of customers that either wish to test from the cloud and those that require infrastructure monitoring along with tests results. In this quickly evolving market, it was very important for OctoPerf to be able to address both of these customer’s requirements as quickly as possible.
On-Premise with Rancher¶
The first step towards this goal was to find a solution to install a local agent for any customer on-premise. And as usual, it had to be easy to deploy and maintain. For instance we did not want it to be a large virtual machine that takes ages to download and/or require advanced linux knowledge to run. That's why in late 2015 and early into 2016 we worked towards integrating OctoPerf with Rancher. Rancher offered a myriad of possibilities in terms of administration, but mostly the ability to install an agent on any customer machine through one command line only.
This made OctoPerf’s on-premise agent very easy to setup for any kind of architecture. If you want more details regarding this particular topic it’s all captured here: on-premise blog post.
CI with Jenkins¶
Another area we focused on is CI with Jenkins, but we didn’t expect our end-users to begin integration so quickly for CI with Jenkins-Hudson most often for automated testing.
This explains why OctoPerf’s second focus area in 2016 was development of a Jenkins plugin that configures everything for end-users.
Read all about it in this OctoPerf blog post: jenkins plugin.
The essence of this solution is that you can run tests and create trending graphs across several builds to analyze performance trends throughout the entire development process.
Browser-based Testing with Selenium Webdriver¶
Whether browser or protocol-based response times are the most accurate is often a subject of great debate. Depending on the situation you may require one over the other, but you know what's even better? Having both at your disposal! For this reason we’ve implemented selenium webdriver test capabilities within OctoPerf. Please note the purpose here is not to generate significant load through Webdriver since accurate response times aren’t guaranteed when you run several instances concurrently, but mixing a low number of selenium webdriver users concurrently to validate the front-end user experience during a load test delivers an enhanced layer of end-to-end performance intelligence.
You can find more details within this post: selenium webdriver.
OctoPerf implemented the Selenium Webdriver test capability as a quick way to measure front-end rendering time while the bulk of load testing is executed via JMeter.
Server-side Monitoring¶
Despite the fact many great monitoring solutions are available in the market today, they don’t always deliver the KPIs you really need are not fully integrated with performance metrics during and after a test to more quickly identify errors and bottlenecks that impact site reliability & speed. We studied several APM solutions to integrate within OctoPerf.
In the end we developed our own monitoring engine because we found most of these solution’s monitoring intervals were rarely good enough to use for performance testing. On top of this installation is often quite complex and time consuming because such tools are dedicated to hosting companies. Another huge benefit with OctoPerf Monitoring is that our existing load agent can be reused to setup monitoring for back-end servers.
OctoPerf’s thoughtful design allows you to quickly and simply add a layer for monitoring on top of any OctoPerf on-premises agent.
Check out all the details in this blog post: infrastructure monitoring.
Update: Since the original release we’ve added more even more monitoring modules for a spectrum of technologies and OctoPerf will continue with such enhancements based on general customer feedback and specific requirements for clients:
What's New Lately¶
We’ve been quite busy since last October to deliver a number of new features within OctoPerf. You may have already noticed some of these enhancements during recents tests you’ve executed with OctoPerf, but let's review all the changes we’ve made below in more detail:
Dataset Automatic Split¶
Have you ever dealt with a website where the user session had to be unique?
Or have a list of records to edit that can't be used twice?
In these cases testing with JMeter implies manually splitting the files across all load generators to ensure every user gets a different value.
This can be a lot of tedious work, which is exactly why this is now taken care of comprehensively within OctoPerf.
Just activate the "Shared" option in your CSV Variable:
Now when you launch the test OctoPerf splits the dataset files across different load generators automatically. This is possible as long as there are enough values in the file for every user to guarantee such values are utilized in a unique fashion. This saves end-users a serious amount of time compared to a manual split and it’s less error-prone.
JSON Path Extractor¶
Many applications use JSON to communicate between client and server.
When you have dynamic values to extract from a response, JSON and RegExp aren’t always the best combination.
RegExp can’t extract any value from a JSON response accurately.
Using the JSON Path Extractor here instead will save you a lot of time, and we further enhanced it with a simple mode:
Just select the node you wish to extract from the JSON Tree.
Of course you can always use any JSON path to extract a specific node utilizing conditions.
As usual with extractors, once created, they can be converted into an automatic correlation rule:
Groovy and External Libs Support¶
OctoPerf already provides robust scripting capabilities thanks to JSR223 logic action implemented a few months back.
But we've recently upgraded it by adding Groovy as the supported language.
Groovy is much more efficient than pure beanshell since it can be compiled:
OctoPerf makes sure it is compiled when running a test to ensure maximum performance.
With this particular update it’s also possible to use the files menu to import your external JAR files:
This adds much more flexibility to any script you create.
These files are sent over to every load generator during tests.
Host File Override¶
We've been asked many times regarding the possibility to override the host file for load generators at runtime.
It’s now possible through the file menu:
Any file named "hosts" will be appended to the end of the load generator’s hosts file.
This also works with user validation since OctoPerf creates a dedicated docker container for each user validation.
NTLM and Kerberos Authentication¶
An important capability for testing inside a corporate network is support for NTLM and Kerberos.
As they are very different authentication schemes, each must be handled on top of the usual authentications.
Within OctoPerf you can set these settings at the server-level. With this way every request for a particular server will utilize these credentials:
Please note that Kerberos requires additional files to work properly: "krb5.conf" and "jaas.conf". Each can be uploaded through the files menu within OctoPerf.
Test By Iterations¶
The only ending condition for a test within OctoPerf is duration. Every user launched will run iterations in a loop until the test duration is finished. (Each run for a single user is called an iteration.) This lack of flexibility is why we added a limit in terms of the number of iterations. This way a test will stop when every user has reached the iteration level configured within the runtime section.
Conclusion¶
We hope you see and experience all of our efforts to significantly enhance OctoPerf’s overall capabilities. You can expect this type of attention to detail and delivery of new features within OctoPerf for years to come.
None of the aforementioned improvements would be possible if wasn’t for all the awesome feedback we receive from our customers, registered users at no charge, and members across the performance testing community worldwide. We can’t thank you all enough for this critical feedback that helps our team create the best possible performance testing platform fully based on JMeter.
Now if you’ll excuse me, we have lots of work to finish still to make OctoPerf even better in 2017. Be sure to follow OctoPerf closely to see what’s new within our platform down the road!