Defining KPI’s in Performance Testing

Defining KPI’s in Performance Testing

When somebody asks “What is testing?”, There are many answers to it. My preferred answer is; “comparing the outputs according to the expected one and checking whether the requirements are fulfilled or not.” That’s an easy answer when it comes to functional testing. What about a non-functional test such as performance testing where requirements are not defined clearly?

Everybody wants their website to respond fast with %100 stability. Your site or app can be fast, but it can misinterpret a request during a high load and order the wrong shoe size for your customer or vice versa. We want both. We want stability and 24/7 availability within acceptable time frames.

During the execution of performance test projects, defining KPIs (Key Performance Indicator) is quite challenging. But this is the factor that determines the quality of the product. There are many KPIs that need to be taken into account. Response time, hit number, error per sec are some of them. Let’s take a look at each one.

Response time

Response time tells us how long a process takes to complete a specific request. In case you are familiar with statistics, we want response times to have a normal distribution when we analyze them in a graph. Most of the requests should have the mean value and some of them should be in the outlier zone.

The x-axis is the number of requests.

Y-axis is the response time.

Most of the requests have the response time between 0.3 and 0.5 sec and mean 0.4. A few of them are outliers. We hope not to see a stable line parallel to X-axis which signals the instability of the system.

Hit Number

We want our requests to hit the server as many times as possible during the performance tests. Whenever a new virtual user is added to the test, we expect the hit number per second to increase. More hits mean faster response. But eventually, hit numbers will become stable after a period, as your server will not be able to respond to all of them in the same manner.

The x-axis is the hit count.

Y-axis is the response time.

We don’t want the response time to increase when the hit number is fixed to a number.

Errors

We don’t expect a system to behave error-prone during a load test. So we expect some errors to occur during the execution. But we want to minimize their frequency. We need to define our error rates in acceptable rates such as:

• We expect %1 of the login request to fail
• None of the “add a product to a cart” requests fail.

That’s why we added KPI functionality in Loadium. From now on, you’ll be able to define your KPIs in your tests and measure your app or websites performance clearly.

In case your KPIs are not met, Loadium will change the status of the test to fail. By doing so, you can integrate your performance test into your continuous delivery pipeline and cut the deployment process in case of a failed KPI.