Performance Tester Diary - Episode 1
This is the first in a series of posts that are going to follow the fictional performance tester Otto Perf. Otto manages all aspects of performance testing for a fictional company OctoCar that specialise in selling used cars.
OctoCar are building a new system to manage the selling of their fleet of cars, and we are going to follow Otto on his journey as he ensures that the new technology platform performs in line with agreed non-functional requirements under peak volumes of load and concurrency.
We wanted to document the journey of performance testing through the delivery lifecycle of a new application from start to finish and look at how the non-functional testing process could be followed as the application is delivered.
Our blog posts normally focus on a singular aspect of the non-functional testing lifecycle and do not always consider the wider picture of non-functional testing, something that this series of posts will do.
Chapters¶
There are 5 chapters in this series with overlapping principles and techniques documented through the journey.
We are conscious that non-functional testing is not always as straightforward as the journey that Otto is going to go on, but we will look to try and introduce some obstacles in his performance testing journey and make the process as much like the real-world as possible.
The chapters will reference back to existing OctoPerf Blog Posts where more detail about the principles and processes will exist.
- Chapter 1: Understanding architecture and business requirements, being involved in application design and creating Non-Functional Requirements and risk assessment.
- Chapter 2: Defining performance testing scope and strategy, understanding load profiles and building performance tests while considering modularisation.
- Chapter 3: Integrating with Continuous Integration and Continuous Delivery in Scrum and adding JMeter tests to push to prod pipelines including their integration with GIT and Jenkins.
- Chapter 4: Using instrumentation to look for bottlenecks, specifically Dynatrace in conjunction with results and trend analysis.
- Chapter 5: Reuse of testing assets for alternative purposes other than performance testing and the introduction of a regression framework for future releases and bug fixes.
Chapter 1¶
The project begins¶
Exciting times at OctoCar as the leadership team had just announced a new programme of work to build a new technology platform to support the re-marketing of their user car fleet. The new platform would be delivered following agile principles with a scrum team that included a dedicated resource to address all non-functional aspects of the programme and would be integrated within the team.
This resource was called Otto, and he was tasked with overseeing all aspects of non-functional testing for this new system. Otto was not new to non-functional testing but had not been solely responsible for all on-functional aspects of a new system before and was excited at the prospect. Otto had been using JMeter for a while and was familiar with the way it worked and understood exactly how he would use this tool to performance test the new technology platform, this would be the best tool to use he concluded.
Where to start¶
Being integrated into a scrum team meant that Otto had the opportunity to be involved in the design process. He had recently read a blog post on the OctoPerf website about how considering performance testing in application design could benefit the programme in the long run and how considering the impact of load at this stage of the process would lead to an application that has been designed to support the volumes that it would be expected to handle in production.
Authors note
There is an article on this very subject in the OctoPerf Blog pages
He also understood that to influence any design decisions he needed to define a set of non-functional requirements. Otto felt that being in the position he was, a new system in the design stage, the very first thing he should do is define a robust set of non-functional requirements. He would then get these requirements agreed with the project team and then he would be a strong position to not only start building tests when the development process starts but would be able to gauge whether the volumes and levels of load were able to be supported very early in the programme lifecycle.
He also remembers reading a Blog Post about Test Driven Development and wondered if the development teams would consider Performance Test Driven Development and thought that would be worth investigating.
Authors note
There is an article on this very subject in the OctoPerf Blog pages
Otto had worked on many programmes of work where non-functional testing was an afterthought and was an activity that was executed at the very end of the programme when it was to late to address any issues relating to performance. This was not going to be the case with this system he thought and went away to define his non-functional requirements.
Non-functional requirements¶
Otto wanted some inspiration for how to build a good set of non-functional requirements and began searching online. Funnily enough he stumbled across another Blog Post by the OctoPerf team on the subject and promptly read this.
Authors note
The article he read can be found in the OctoPerf Blog pages
Otto understood that non-functional requirements were loosely defined as the 'ilities' and after reading the article he was confident that he understood these correctly. He was intrigued by the concept of making sure your non-functional requirements are testable by combining individual requirements to ensure response time is only measurable when under the correct level of load with the infrastructure utilisation withing agreed tolerances. Otto decided to make this a guiding principle of his non-functional requirements.
Technical and business resources, allocated to a project, are often involved in the designing principles of a new technology platform.
The reason for this is that the purpose of what is being built will be to provide a technology platform to improve, streamline or update the way the organisation works. As part of this process business and technical requirement are considered and defined. What Otto could not find in any of the current requirements was any indication of expected volumes and infrastructure sizing. Otto assumed that this is common, and he was correct.
After speaking with the business resources allocated to the programme, he felt that they had all learnt something:
Otto had a clear indication of expected volumes at peak times, level of concurrency that can be expected, seasonal peaks and expectations around what constitutes an acceptable response time for customer facing transactions.
The business users, in turn, had lots of information on how the application would be performance tested, measured and results documented. And more importantly felt included in the process and definition of what is considered good performance. Otto had the same conversation with the architects and technical leads where the focus was more around how the platform would scale and how the infrastructure would be sized initially.
This also gave Otto the opportunity to talk to the technical team about how data could be seeded in the test environments and whether he was likely to get a test environment that mirrored production to test on. Otto spent some time using all the information he had gathered from these sessions and creating a set of non-functional requirements along with a definition of how these would be tested.
After a few rounds of reviewing and refining these requirements with both the business and technical resources Otto had a 88complete and comprehensive set of non-functional requirements as well as an agreed way of testing these**. Otto was looking forward to the application design process to see how performance could influence this.
Risk assessment¶
Otto was aware of the concept of risk assessment as he had also found another article on the subject.
Authors note
The article he read can be found in the OctoPerf Blog pages
There were many requirements that had been defined and to test them all, especially when timescales are tight, is not always possible. He was glad he had documented them all as a record of what should be tested if time permitted but he wanted a way of prioritising these.
Using a standard risk assessment matrix of impact v’s likelihood and again using both the business and technical resources he had built a relationship with he was able to agree on a priority order that he would use throughout the programme.
Combining a good non-functional requirement process along with a simple but effective risk assessment gives all involved in the programme a very clear view of what we are trying to achieve from a non-functional perspective thought Otto.
Application design¶
Otto attended several of the design session and found it unusual that the Quality Engineering team were not generally involved in this part of the process. He felt sure that it would not only benefit the Quality Engineers but would also provide another viewpoint on the design principles from an alternative perspective. Using the non-functional requirements that he had defined Otto was able to actively join in the discussion on many aspects of the applications technology stack and how it should be sized.
Because the principal members of the design team had been involved in the definition of the non-functional requirements, they were receptive to the idea of using them to assist in the design process. As well as being able to help in the design Otto also found that he learnt a lot about how the system will be monitored and deployed to and the technology being used to host it. He was able to understand what type of database would be used and started to understand how it would use message queues and Kafka to distribute messages around the system. Otto was able to use this information to further update the non-functional requirements with the knowledge he now had.
Because he was currently unfamiliar with the database technology and the tool being used to instrument the application, he made a mental note to go and research these technologies to further understand more about them. Otto found some time to spend with the wider Quality Engineering team to bring them up to speed with what he had found out and they agreed that their involvement in this part of the process can be hugely beneficial.
Lessons learnt¶
Otto decided that he would make some notes on what he had learnt in the early stages of the programme, these are his notes.
- Involving and speaking with as many of the project and business team as early in the process is an advantageous thing to do.
- It makes the principle of performance and the wider implications of non-functional testing something that everyone understands and that everyone is committed to achieving.
- Because of the transparent nature of the processes he had followed, and before a single line of code has been written the whole programme is familiar with the non-functional requirements the system needs to achieve and this will be in the minds of the developers and technical principles as the applications begin to be developed.
- Use the non-functional in the application design sessions to ensure that consideration is given to ensuring that performance and scalability are factors considered when building an application.
Conclusion¶
The way that Otto made non-functional testing a subject in the forefront of everyone’s minds very early in the programme is principle that you should try and adopt.
Clearly this is not always possible if the involvement of the non-functional test resource is not from the very beginning of the programme but as soon as possible would also be beneficial.
In our opinion non-functional testing suffers when the impact and implication of poor performance are not understood and when systems are tested against volumes that are unrealistic with response time target equally unrealistic. Transparency around this is a huge benefit.
Join us soon for Chapter 2 in the performance testing adventures of Otto Perf.