Risk Assessment In Performance Testing
Performance Testing coverage needs to be defined from the application under tests functional requirements and this does not change regardless of whether you are following an Agile or a more Planned approach.
The Risk Assessment process defines what performance testing needs to be executed and the order in which this should be approached, the reason we use the functional test requirements for definition of coverage as these functional requirements define what the system does whereas the non-functional requirements are what you use to measure your tests against when you execute them.
As discussed the Risk Assessment is a way of identifying where your performance testing effort should be focussed and prioritising the order in which your performance tests are written and therefore executed so you ensure that you are focussing on the riskiest, from a performance perspective, aspects of the system first; we will expand on what we mean by riskiest later in this post.
A Risks Assessment should be done as early in the project or sprint, if you are working in an Agile methodology, as is possible to maximise its benefits.
The first thing we will discuss is the differences between Agile and Non-Agile Risk Assessment as this is an important concept but only as far as what is assessed and when, the principles of how and why we carry out a risk assessment will become clear as you read this post and remain the same regardless of your approach to development.
Agile vs Non-Agile¶
It is worth mentioning this at the beginning of this post as it will alter how you perform your Risk Assessment.
In a more traditional planned approach to application development you will have a full set of application requirements available at the start of the project and a Performance Risk Assessment can be carried out on the whole application.
If you are working in a programme following an Agile methodology then you will only have visibility of the slice being worked on and even then you may only have access to the backlog for the sprint you are about to start which means you have to Performance Risk Assess the part of the application you have visibility of.
The only real difference in how a Risk Assessment is carried out between the two approaches is that in an Agile way of working you need to be aware of what you agreed in previous sprints to ensure consistency in your approach which when working on a full set of requirements is not something you need to worry about.
What is a Risk¶
In order to perform a Performance Risk Assessment, we need to determine what our Performance Risks are and how they must align with our non-functional requirements.
We will take several simple examples from start to finish so you fully understand how the functional requirements and the non-functional requirements need to be used for a Risk to be created.
Once we understand how we define Risk we can move on to how we assess them.
Let’s say we have a customer search function in our application we would want to build Performance Risks around this part of the application.
A simple risk might be:
- The customer search functionality will not perform.
This Risk is a bit ambiguous because we have not defined what performance is, lets refine our Risk:
- The customer search functionality will exceed 2000 milliseconds when returning all search results.
This is better as it is measurable with the relationship back to a non-functional requirement, in our example being the response time of the function.
Let’s look at other Risks we may define for our search functionality.
-
The customer search functionality when performing 5 concurrent searches will exceed 2000 milliseconds,
-
The customer search functionality will consume more that 40% CPU when run at peak volumes,
-
The customer search functionality will not release memory on completion.
As you can see, a Risk is always a negative thing rather than a positive thing as you need to understand the implications of the system not performing in order to focus your performance testing effort in the right place; for example
- The customer search functionality will return all search results within
2000 milliseconds.
is positive and therefore not a Risk.
Your whole application or the slice you know about should have its Risks defined this way.
Assessing our Risks¶
So, we assess our Risks by determining what the Likelihood of the Risk happening versus the Impact of the Risk happening.
In performance testing terms this means assessing each feature of your application under test in terms of the Likelihood of a feature not performing versus the Impact on the business of the feature not performing.
A simple Risk Assessment template looks like:
You can add complexity to your Risk Assessment template but sometimes it is better to focus on the Risks themselves rather that overcomplicate the analysis.
The scores in the coloured boxes are derived from multiplying the Impact value by the Likelihood value with a score less than 3 considered Low Risk, greater than 5 High Risk and anything in between a Medium Risk.
The process of scoring risks and who should be involved in the process is discussed in the next section.
It is no more complicated than that, let’s look at our made-up risks that we defined earlier and look at scoring these so you can get a feel for the way we might display them in a spreadsheet or document.
| No | Risk | Comments | Impact | Likelihood | Score |
|---|---|---|---|---|---|
| 1 | The customer search functionality will exceed 2000 milliseconds when returning all search results. | The impact of slow searches is that the call centers will keep the callers waiting. | 3 | 1 | 3 |
| 2 | The customer search functionality when performing 5 concurrent searches will exceed 2000 milliseconds. | During peak call volumes we do not want to see a degradation in performance of searches which will impact the call centers. | 3 | 3 | 9 |
| 3 | The customer search functionality will consume more that 40% CPU when run at peak volumes. | If CPU consumption is high then we will be unable to support full load should one of our servers fails. | 2 | 1 | 2 |
| 4 | The customer search functionality will not release memory on completion. | If memory consumption is not constant and grows then we will see performance issues over time. | 3 | 2 | 6 |
Clearly this is an example but hopefully it gives you an insight into how you might approach your Risk Assessment.
Risk Assessment Process and Audience¶
So we have looked at how to define our Risks and we have look at a template we could use to Assess our Risks and we will now look at the process of Risk Assessment and the audience you need to help you with this process.
You need to categorise what is low, medium and high for likelihood and impact before you start as using a gut feeling or a guess is not a sensible way to approach this exercise.
Let’s look at some of the ways you could categorise these although these will undoubtedly be different for each organisation and application under test.
We will start with Impact:
| Low impact | Medium impact | High impact |
|---|---|---|
| A work around is available | Will cause an overhead on the business teams using the application | Will cause a loss of revenue |
| It’s an infrequently used piece of functionality | Will lead to poor customer service | Will damage your company’s reputation |
| The functionality is not part of a customer facing role | Will lead to a loss of customers data |
As we can see the Impacts are focussed on how the Risk will affect the users of the application whereas the Likelihood, which we will discuss next is more focussed on the technical attributes of the application.
| Low likelihood | Medium Likelihood | High Likelihood |
|---|---|---|
| It’s a simple piece of out of the box functionality | Multiple systems interacting with each other | Complex code |
| The code is simple | Relies on Synchronous calls to Legacy systems | Multiple concurrent application processes |
| No complex calculations or database queries | Complicated technology stack | Large dataset to process |
These are examples of how you might approach the scoring and will hopefully provide you with a starting point for any scoring systems you put in place.
The process of scoring risks is normally done in an interactive way where each Risk is discussed and scored as part of a collective process and the audience to these sessions is important.
You need to include people from the programmes that are both business and technically focussed as both these qualities are required in order to complete a good assessment of your Risks.
Include functional QA’s, business Analysts, Project Managers, Programme Manager, in fact invite as many people as possible; this is all about transparency and all members of the programme feeling that collectively you are deciding on the scope and priority of the performance testing. This will help in the long run when signing off the application performance results as some of the approvers, if not all, would have been involved in this part of the process.
What happens if all Risks are scored high¶
It is possible that all the Risk you have identified are scored high and under these circumstances you may still need to prioritise as time is short or there are regulatory deadlines that cannot be missed.
If this happens you first need to evaluate if you need to hold a further Risk Assessment session to try and re-score but if this is not possible or not appropriate then you should look to work out if multiple Risks can be covered with a single test and look to focus on these first with the aim being to get as much coverage as you can with the minimum number of physical tests.
Obviously if time is not an issue then your scope is clearly all Risks must be tested.
Can you re-evaluate your risks¶
Risks change and they should be regularly checked, the result of one test on one part of the system may, depending on whether the results are positive or negative, change the perception of Risk on other pieces of functionality.
As each test is executed your risks should be re-evaluated and revised and all parties involved in the initial assessment process kept informed.
If you are working on a programme following Agile principles, then you will have to run Risk Assessments and determine the impact of the stories in the new Sprint or Slice as well as understanding how these fit with your current Risks.
How should your process evolve¶
There is not a right way to Risk Assess, whether for performance testing or functional testing, and therefore it is important to determine what works for you and your organisation.
If you feel the process you have adopted or used previously was not right, then change it.
The critical thing is to do a Risk Assessment or some sort and to make the scoring and outcomes transparent; a Risk Assessment process that did not quite deliver on a programme is better than not even attempting one.
The hope is that you will be encouraged to try the principles and practises that we discuss here and that they give you confidence to start if you do not have a process or refine if you find any of the information in this Blog Post useful.
Are there Tools I can use¶
There are tools that allow you to build a Risk Assessment matrix and track coverage and relationships between Tests and Risks and even Requirements and a simple search online can provide many examples. These tools can be expensive unless they are shipped with other products that your organisation uses.
I think that you could use a spreadsheet as this is just as good and the complexities of your spreadsheet are only bound by your ambition. Basically it can be as simple or complex as you want to make it.
Keep a risk register¶
It is worth keeping a register of previous Risks from previous projects and programmes as firstly it will allow you to adapt and improve your techniques as you run more Risk Assessments and you may find that should a new enhancement be added to an existing application you can re-use you previous Risk Assessments as a starting point for the new piece of work.
Conclusion¶
Risk Assessments make your performance testing coverage transparent to all key project stakeholders and makes them feel involved in the process.
It will give you a clear order of tests you need to create and run and will help you prioritise in the event of tight timescales.
It is a useful tool to use as a performance tester.