SNCF - Case study
Most of you have already recognized the name SNCF, it is obviously one that is hard to miss when you live in France. But for everybody else, allow me to do a quick reminder of what SNCF stands for.
The Société Nationale des Chemins de fer Français (Chemin de fer, literally, 'path of iron', means railway) is France's national state-owned railway company. It operates 32,000 km (20,000 mi) of route and in 2017 had €33.5 billion of sales in 120 countries. The SNCF Group employs more than 260,000 people.
source: Wikipedia
Lately, SNCF's IT strategy could be summarized as follows:
- Have state-of-the-art, multi-cloud, application execution capabilities,
- Work as a business partner with hand-picked software vendors to help them grow and learn from a real life use case.
This means re-thinking the strategy in many areas, including performance testing.
The factory provides assets to manage the entire application lifecycle. Julien was looking for
a solution that could fit in the factory while still answering to a large variety of requirements.
To help with his task, Julien was assisted by Joaquin De Ibar Aguado who took the role of project manager on the migration to OctoPerf. Joaquin would help integrate OctoPerf in the factory as well as migrate a few key projects as a proof of concept.
Initial situation¶
But first let's take a step back and assess the initial situation in more details.
In terms of organization SNCF is one large entity composed of many IT departments. Each one of them is free to choose their own toolset. Due to this, prior to OctoPerf, several tools/offers existed:
- An offer around JMeter,
- A Performance Center platform,
- Several smaller platforms built for custom purposes.
Because of that choosing OctoPerf as the only tool would be a way to limit shadow IT and centralize the practices and methodology. The idea is to propose a single offer simple to use & integrate in every project. In due term this should improve the performance testing coverage by making it easier to setup.
Another aspect is the integration into the DevOps toolchain of the software factory. That would prevent a lot of manual work and make the test process even easier.
Setup¶
OctoPerf's support team and SNCF's team worked closely together to accompany all projects toward migrating into OctoPerf. At first the JMeter scripts had to be adapted to fit perfectly into OctoPerf's model. This created some confusion initially but everyone quickly realized that the changes were minimal and the benefits were important. The idea with OctoPerf is to find the right balance between a large number of functionalities and a much better ease of use.
For instance the runtime settings have to be redefined initially, but since you can create many scenarios and save them for later use, it quickly becomes a time saver:
This pilot phase was conducted with a dozen of projects and proved very successful. Feedbacks were very positive and the time saved using OctoPerf was appreciated. This also allowed each IT department to train some of their testers on OctoPerf and start building templates.
Result¶
Now OctoPerf is being used by 30-50 regular users, 2 platform administrators have been appointed to deal with first level support and authorization of the various teams across the workspaces and load agents inside OctoPerf.
On the OctoPerf platform several third party IT services companies and employees of SNCF collaborate to perform performance tests. Since OctoPerf natively allows to split load agents between projects by creating dedicated docker containers for each test, it gives flexibility when a lot of users are testing at the same time. Typically the test infrastructure was reduced from 10 servers to only two which means a significant decrease in cost.
Also, the fact that the OctoPerf server is hosted as an external service by OctoPerf also offers even more flexibility, in particular when a lot of people suddenly have to work from home.
If you're curious about this and all the other possibilities (SSO, cloud integration, etc...), we've already covered most of them in a different article on this blog.
Future¶
As part of the collaboration with SNCF, we developed our Azure integration and a lot of other smaller features to make OctoPerf easier to manage on a daily basis. We want to continue this trend, for instance to make it even easier to start instances on demand inside the SNCF private cloud.
Another topic is the integration of monitoring metrics from Datadog into OctoPerf, this way OctoPerf will become the performance metrics repository where all metrics can be compared and cross-checked.
There's also more than 100 applications to test from OctoPerf each one of these having its specificities. Including some we've already worked on:
- a token to share among all users at runtime,
- modular design through fragments and link actions,
- etc...
Of course the plan is to keep going and to find ways to answer all the requirements that also make sense for OctoPerf as a whole. This way we can make OctoPerf even better for SNCF and all our other customers!