There are many different testing techniques you can apply to your application under test (AUT). Some tests are categorized under functional tests whereas some of them are classified as non-functional tests. Load Testing is one of the most important non-functional testing disciplines. There are many names defining that activity, some call it “Load Testing”, some call it “Stress Testing”. While they all have different approaches while executing the tests, the general purpose is mostly the same. They aim to define how the software application behaves under specific load simultaneously or a continuous amount of time. By doing so we find the bottlenecks of the application and improve the performance of it.
Since the rise of mobile devices and the spread of the internet, the user number of any application has increased dramatically. That is a problem. How are we going to test the performance of the application? This is where Cloud Performance Testing gets in our life. Thanks to tech giants like Amazon, Google or Microsoft, we now have cloud technologies that simplify our life in every aspect of software development lifecycle. Cloud computing helps IT teams to provide on-demand availability of computer system resources and enables them to use computing power without having the actual hardware. So those technologies allowed test engineers to apply more serious and beneficial performance tests. There are two types of performance testing on the cloud.
- SaaS Cloud Testing
- On Premise Load Testing
What Is Cloud Load Testing?
If we need to describe what a cloud is, we can easily say “somebody else’s computer”. In cloud load testing, you execute your test scripts in a company’s privately owned machines. Basically, you outsource anything related to physical devices like storage, maintenance, security and any updates that need to be applied to them.
What Is On-Premise Load Testing?
On contrary to Cloud testing, on-premise load testing enables you to execute your performance test scripts on your physical devices which can be situated in a data center. Then, you would need to deal with all the physical devices we mentioned previously.
What Are The Deciding Factors?
Goal
The main goal of both approaches is to make test engineers’ lives easier while doing performance testing because of the huge CPU, memory and device needs.
Audience
One of the challenges of Load test is trying to understand your user base and creating test scripts accordingly. Every user has a unique behavior model and most importantly a different browser and device. There are many performances testing tools in the market.
First group of tools allow you to implement a test script on the network layer by emulating the protocols that you test and the second group allows you to script both on the network layer and on UI level. JMeter, Gatling or Locust are the most known tools of the first group. By using those tools you can emulate network calls. You can define scenarios while changing user agents like a real user. Second group of tools are based on UI level interaction so you don’t have to deal with network layers, you can develop Selenium scripts and use real browsers during your tests. For more information about defining KPIs
- Budget
They both have a cost for your company. Cloud load testing requires a small amount of cash at the beginning, but you might be paying more because of the yearly subscriptions in case you won’t take advantage during the whole year. On the contrary, on-premise load testing requires more money initially as you’ll need hardware and software to set up a performance testing environment. But in the end, it might be beneficial in the long run as it’s up to you to use the environment or not. No need to pay any subscription fee for the times you don’t use it.
Virtual user capacity is one of the key factors in Load Testing. When performing a load test, the biggest debate among teams is always “virtual users vs transaction”. So, the question is; “Is it important to have as many Virtual Users as we have in production?” or “should we create the load by creating the transaction traffic like in production?”. It all depends on the nature of your application.
In case you host an application where sessions are dedicated to users and one user can’t have more than one session, we need to stick with Virtual User strategy. Think about WhatsApp’s web application. You can’t have more than one tab open at the same time.On contrary there are other types of applications where we can use one user for different purposes. For that kind of application, we can choose to follow “transaction strategy”. A newspaper website can be a good example of this. One user can open as many tabs as possible and do many things in all those different tabs at the same time.
So while one app can create “N requests” with one user, the other app can create two, three times more requests than the other user. According to these variables, we can choose how many virtual users we are going to need while testing the application.
This decision will shape our performance testing selection. In case we need more virtual users, it might make sense to use a cloud provider to easily scale up/down horizontally or vertically.
But VU’s become more important when it correlates with geographical distribution. Nowadays many applications are used by people all over the world. If all your load simulators are in one place, it’s not possible to properly simulate the production load. Think about hosting your app in Berlin and simulating a load for different users from the US and France. Network hops count, latency and of course response times will be different. Web apps that have been accessed by anyone require load tests from different regions.
Testing Skill Set
On both approaches most of the things about testing concepts don’t change. But in case you have an on-premise solution, your team will also be responsible from the testing tool environment, then it’s very important to have team members that have the understanding of how to maintain virtual machines, provisioning and security. Most of the time a DevOps guy will be responsible for those, but your test team should be aware of what’s going on in case of any problems. If you need help about Load testing, we are here to help!
Test Assets Availability
Test asset is your web application under test. When we do an on-premise performance test, it will be so smooth to configure our Load Balancer, our security policies. The reason is that you know from which IPs the load will be generated. So, your security protocols can be configured to allow the requests coming from those IPs. In a SaaS testing solution, you’ll never know which IPs will be used and so your security team might need to grant more privilege than it’s needed. On the Load Balancer’s side, if load is generated from many different regions, it is easier for the load balancer to dispatch the request to different servers. This will be a more realistic test. But in an on-premise solution many of the requests will have the same source so it might be hard to configure your load balancer.
Timeline and Urgency, etc.
When you want to host an on-premise load testing platform on your hardwares, you need to spend a lot of time for the first installation and configuration. All software and hardware should be set up by our own resources.
For some tests you will need a small cluster but for bigger load tests, you will need to scale up at the beginning or during the test. This is something not easy to achieve when you have an on-premise load testing solution. But this is much easier when you have a cloud-based load testing solution.
Use Case and Script Complexities
The only thing that does not differ when we need to use a cloud based or on-premise load testing solution is the scripts. The scripts are mostly the same because we deal with different protocols to make calls to the server. Those protocols don’t care how you conduct your performance test.
The only thing that might differ can be managing the test data. In case you use a cloud-based solution, and you need to access an internal database, you will need to manage the access permissions. But with an on-premise load testing tool, that won’t be an issue at all.
Pro’s and Con’s of On-Premise Load Testing
Let’s start with Pro’s
This may simply be a business decision. For some specific sectors with rigid compliance and regulatory requirements running the tests on-premise may be a necessity. This way, the ownership of the load generators enables you to control who has access to the data. An on-premise environment even allows you to lock down access, if necessary, while this is not possible with cloud load testing. If you choose to proceed with cloud load testing, you are advised to have security arrangements with the cloud provider that you choose, detailing who can see or access the test data or identifying client information.
- Lower cost for load tests for limited users and one location.
- Better control over the environment as you own the hardware with full operational access.
- Better data security as all test scripts will remain confidential.
- Good for internal applications.
Now the Con’s
The ownership of load generators has understandably a higher up-front cost, which makes it infeasible for smaller projects or businesses. Since larger projects need concurrent users, this means more machines, the more licenses you need for your testing tool, and more time to recover your initial investment. This makes cloud testing more attractive for small and medium enterprises.
- Responsibility over all aspects of the maintenance of the load generators.
- Big initial cost along with ongoing maintenance cost which makes it a better investment for large enterprises.
- Requires manpower and relevant skills to manage your own on-premises testing environment.
- The maintenance can become complex.
What Makes Loadium On-Premise Load Testing Solution Different?
100% Docker Compatible
Loadium on-premise load generator works 100% compatible with Docker. By simply installing Docker, you can run and manage Loadium on-premise agents.
Set Up A Node Less Than 1 Minute
Loadium on-premise load generator performs a faster installation than other tools. Loadium allows setting up a node less than 1 minute. Don’t wait a node setting up for 30 minutes.
Highly Scalable
Loadium on-premise load generator allows increasing or decreasing the on-site nodes according to the capacity of the machine easily.
Conclusion
Both on-premise load testing and cloud-based load testing are great solutions. Thanks to the growth in cloud-based technologies, adopting a cloud-based performance test environment is very easy. It has many advantages and disadvantages as on-premise load testing. That’s why it’s crucial to understand your company’s and projects’ needs from different perspectives. Then you can start with the right tooling and benefit from it.
