Real-time analytics with ElasticSearch
When we first started to think about real-time analytics, we though that our previous experience in load testing would help us to get quickly to something working fine. It always seems easy to rethink about a problem you already solved in the past. We were totally wrong.
We wanted to provide our users a completely new analytics experience when load testing their website. Our requirements are :
- Real-time: users should not wait the end of the test to analyze results,
- Blazingly Fast: users should experience sub-second response times, regardless of the number of concurrent virtual users,
- Scalable: we must be able to scale horizontally as the number of users grows,
- Reliable: the analytics system should be redundant to provide fault tolerance and avoid outage,
- Testable: we should be able to easily verify through unit testing that the analytics system is working correctly,
- Open-source: we don't want to build this by ourselves. We believe that it's not our job. We don't want to waste our time reinventing the wheel.
We placed the bar very high. If you want to reach the moon, shouldn't you somehow aim for the stars?