Performance Response Times
When performance testing you need a set of requirements to measure your response times against. When defining these you should do so with your end users or business teams.
It is relatively easy to predict volumes, load and users that will use your application as you will no doubt have some data based on your current systems. It is a lot harder to agree on what the response times of your application should be. Without this critical measure we really cannot say for certain that an application or service performs. The reason being that we are not measuring the response time under load against a value that has been agreed upfront.
We need to be able to answer these questions:
-
At what point does a response time become unacceptable?
-
How can response times be defined in the requirements gathering stage?
-
How can we ensure there is a measure to test against?
We are going to look in this post at how we might categorise response times based on the interaction model with the end-user. And use these metrics to build response time requirements as part of your wider non-functional requirements.
Is there a set formula¶
There is no set generic industry standard for performance response times and there is very little been written about the subject. It really depends on your application and the audience that will be using it.
The following summarises research conducted in the late 1980s that attempted to map end-user productivity to response times. It looks at how productive the end users could be based on the speed of the technology platform they we using.While the research is old it provides a useful set of conclusions that to a degree are still relevant today.
Let’s look at the findings and we will discuss further in the next section.
Greater than 15 seconds¶
This rules out conversational interaction. For certain types of applications, certain types of end users may be content to sit at a terminal for more than 15 seconds waiting for the answer to a single simple inquiry.
However, to the busy call-centre operator or futures trader, delays of more than 15 seconds may seem intolerable.
If such delays could occur, the system should be designed so that the end user can turn to other activities and request the response at some later time.
Greater than 4 seconds¶
These delays are generally too long for a conversation, requiring the end user to retain information in short-term memory (the end user's memory, not the computer's!).
Such delays would inhibit problem-solving activity and frustrate data entry. However, after the completion of a transaction, delays of 4 to 15 seconds can be tolerated.
2 to 4 seconds¶
A delay longer than 2 seconds can be inhibiting to operations that demand a high level of concentration.
A wait of 2 to 4 seconds can seem surprisingly long when the end user is absorbed and emotionally committed to completing the task at hand.
Again, a delay in this range may be acceptable after a minor closure. It may be acceptable to make purchasers wait 2 to 4 seconds after typing in their address and credit card number, but not at an earlier stage when they may be comparing various product features.
Less than 2 seconds¶
When the application user has to remember information throughout several responses, the response time must be short.
The more detailed the information to be remembered, the greater the need for responses of less than 2 seconds.
Thus, for complex activities, such as browsing products that vary along multiple dimensions, 2 seconds represents an important response-time limit.
Subsecond response time¶
Certain types of thought-intensive work (such as writing a book), especially with applications rich in graphics, require very short response times to maintain end users' interest and attention for long periods of time.
An artist dragging an image to another location must be able to act instantly on his next creative thought.
Decisecond response time¶
A response to pressing a key (e.g., seeing the character displayed on the screen) or to clicking a screen object with a mouse must be almost instantaneous: less than 0.1 second after the action.
Many computer games require extremely fast interaction.
Is the above still relevant?¶
Based upon analysis carried out from a productivity perspective we can see that this is still relevant. A threshold of 2 seconds was the recommendation for applications that interact with users. This would still seem a sensible and again might be a response time to use as your requirements for all user-based interactions.
When this analysis was taking place the concept of service-based architecture and cloud computing was not something that would have been considered. With this in mind services interaction would fall in the sub-second response time category.
We would also need to consider asynchronous transactions where the application is processing after a response has been received. The analysis above would seem to indicate that the 15 second category is the one that asynchronous transactions fall into.
The analysis above would then seem to be a great starting point for defining your non-functional response time requirements. It helps define it based on so real analysis rather than assumption of guess work.
Our take on response time requirements¶
Clearly, as we have already stated, each organisation and application is different. Having a standard set of response time requirements is not the right idea, having boundaries that you can work in is a more sensible. Categorising your business functionality or user journeys this way helps with requirement creation.
Batch response times need consideration as well which is something that is missing from here. Using the same principles you can categorise your batch response times based on how long your batch window is verses how may batches need to be run. You can also look to include their relative importance and complexity into your thinking.
Conclusion¶
Response times are difficult to define and even harder to get agreement on with the end-users. Using a category-based system is useful and scientific enough for the business and development teams to agree on. They provide a way, if needed, of prioritising your testing focus by prioritising the business transactions with the more aggressive response times.