---
title: Performance testing and objectives
url: https://blog.octoperf.com/performance-testing-and-objectives/
date: '2015-10-29'
authors:
- gbetaillouloux
categories:
- Load-Testing
tags:
- Methodology
- Investment
- Response Time
description: How defining clear objectives will drive your performance testing campaign?
---

# Performance testing and objectives

There's no denying the importance of a performance test campaign in the quality assurance process.
But I have lost count of customers requesting load test with no specific objective (or just "improve the performance").
**Running a test campaign with no objective in mind is quite risky**.
You might spend a lot of efforts on useless improvements whereas the critical pain points remain.

## Endless tuning

![Plan](https://blog.octoperf.com/img/blog/performance-testing-and-objectives/plan.jpg)

**I worked on performance campaigns with no objective defined a couple of times.**
Usually, this implies running tests until you run out of time/budget and optimizing whatever you can.
It does not mean these campaigns were not successful, but **they probably cost a lot more money than they should have**.


First, you often end up with a larger infrastructure than needed which of course does not comes for free.

> I once witnessed a customer purchase two large machines as application servers a couple of days before the go live.
> These machines allowed the application to handle 4 times the standard load.
> But the application was never expected to have so many users connected.
> In the end a lot of money got spent that way.

Then there is the cost of the tests themselves.
Even if you use open source tools you have to consider :

- The test infrastructure,
- The performance tester(s),
- All the contributors.

When you add up all of this you will have the impression that performance testing is expensive.
A lot of companies would like to have us think that way.
But **if you have a precise objective in mind you can control the costs efficiently**.

## Defining an objective

![objective dog](https://blog.octoperf.com/img/blog/performance-testing-and-objectives/objective-dog.jpg)

You can't blame a newcomer for not having a precise objective.
**It is the role of the tester to help define the objective** by asking the right questions or by giving examples.

Common objectives are :

- Response times (or error rate) under *X* seconds (or *XX* percent) at standard load.
- Validate the load balancing/failover mechanism
- Confirm that *XXX* users can be connected simultaneously
- Check that the background tasks don't have an impact on the response times

This topic should be **addressed as soon as possible because it will have an impact on the whole campaign**.

## Risk based testing

![Risks](https://blog.octoperf.com/img/blog/performance-testing-and-objectives/risk.jpg)

**Performance testing is all about covering risks**.
Testing the load balancing covers the risk of a load balancing failure in production.
With this in mind, to properly define objectives you have to **consider the risks you would like to be covered**.
Cost wise, you won't be able to cover all of them so try to focus on what matters the most.

For example, let's assume you test a website which will be accessed worldwide and 24h/24h.
Amongst other things, you must run a 24h test (soak test) to assess if server's memory and network capabilities are stable over a long period of time.

It is all about **assessing the circumstances that would put your system at risk and addressing them with a test**.
This is especially difficult for a first deployment where you have no statistics about the application usage.

## What to do with my objectives

Now that you **assessed the risks and defined objectives accordingly**, you will be able to prepare for the rest of your campaign.
At this stage, you should be able to determine :

- The virtual user profiles to create and their complexity.
- The number of tests to be launched.
- The expertise required to prepare and analyze the tests.

My point is that defining objectives is critical to get a clear view on the delays and work to be done.
This will also tell you **if your tests can fit in the timeline and budget of the project**.

## In short

**Establish an objective first**, this will help you assess the complexity of the test campaign.

To help you establish objectives, think about :

- Common objectives such as **response times and number of users supported**.
- **Risk based testing and your desired coverage**.
- **Your budget**.