When it comes to performance testing, there are not so many different tools that has been mature enough to use in software performance testing projects. In this article we decided to talk about two of the oldest performance testing tools on the market. Those two tools are Apache JMeter and LoadRunner. And we will briefly look at the details of JMeter vs LoadRunner.
Let’s give some information about LoadRunner for those who have never heard of it.
Apache JMeter and LoadRunner
What is Apache JMeter?
Apache JMeter is an open-source Apache project that helps IT teams to do load testing on their applications. It has its own script recorder, test result analyzer plug-ins and many cool features.
What is LoadRunner?
LoadRunner is a software performance testing tool developed by HP. But after the acquisition by HP, Load Runner became a Micro Focus tool.
Load Runner is one of the oldest performance testing tool in the market. It is used to test applications’ behavior and performance under load. You can create your performance test scripts by its recorder or do some coding from scratch by using its own development language. Then of course analyze test results by its reporting tools. As Load Runner is on the market for so long, it has support to so many different technologies.
Pros and Cons
Pros & Cons of Apache JMeter
First, let’s talk about the pros.
JMeter is a very light weight tool which supports tons of different communication protocols like HTTP, FTP, JDBC, etc. It has no other rivals that can compete with JMeter. You don’t need to worry if you can use JMeter in a performance project or not. It will be able to execute any kind of performance test.
Then it’s free, you do not need to pay anything to execute a performance test even with 10.000 users. Of course, you need to arrange your execution environment according to your specs, but the limit is the sky. In case you cannot afford to create those environments, you have many Cloud providers like Loadium that will help you generate thousands of virtual users in a fleeting time.
You can extract from various kinds of responses like xml, json, html by using its own extractor plugins.
You can extend the tool in case you have Java knowhow.
One of the cons is its memory consumption when you use it in GUI mode. It is likely to crash under heavy load but thanks to its command line interface that disadvantage disappears. It might be hard to test dynamic content and Ajax based applications, but all those problems are achievable by scripting. Finally, built-in reporting is a bit weak. You need to install 3rd party plugins or use cloud providers to execute your scripts to have more user-friendly reports.
Pros & Cons of LoadRunner
As an enterprise, LoadRunner comes with amazing features. Firstly, the look and feel of the tool is very modern compared to JMeter. Anyone who uses a desktop application can get familiar with the usage of LoadRunner. Secondly, the most important feature is stability.
LoadRunner is very stable when we want to create load from a local machine compared to Apache JMeter. It has built-in cloud execution support.
Reporting is detailed with a report that enables you to deep dive into so many metrics.
Cons of LoadRunner is firstly the platform support. Mac users can’t use it. Then you need to pay a lot to acquire the tool. And most importantly you will need some time to get used to its scripting language.
6 Most Important Factors to Decide, JMeter vs LoadRunner?
Availability
JMeter runs in any environment like Windows, MacOSX and any Linux distribution. On the contrary LoadRunner works only on PCs and Linux systems. So if you are working for a company that distributes Macs to its IT teams, you won’t be able to use LoadRunner. As a result, Apache JMeter is accessible to anyone who uses a regular PC.
One of the most important parts is that JMeter is a very lightweight tool. You can download it in a few seconds and start using it in a few minutes after the installation. Of course, there are plugins that you might want to install so you can get more out of JMeter, but the standalone package already offers a lot for a load test. In the meantime, Load Runner is a very heavy tool to install. It has so many different installation packages with different features.
So, the onboarding process becomes quite easy for a performance test engineer.
Scripting and Test Cases Implementation
Both tools provide record and play functionality. That is an effective way to start performance testing. But eventually you will need to do custom scripting to enhance your test cases with test data and more importantly you need to manipulate the CSRF, authentication tokens for dynamic apps.
JMeter has so many components helping you to extract, manipulate the data so you can use those to enrich your scripts. 90 percent of the time, those components will be enough and all you will need to do is drag and drop those components and configure them.
But in LoadRunner, there are no such drag and drop mechanisms. Most of the time you will need to do custom coding with LoadRunner’s own development language.
Load Generation Capability and Execution
JMeter is a very structured tool. You can easily identify which script does what kind of an action with how many users. After configuring the scenarios by scripting or recording, all you need to do is to use the command line interface to run the virtual user generators. Then you will need to configure your JMeter installation to recognize the IP of those machines by editing a configuration file. The problematic part, when you add a new user generator you need to edit that configuration file once again manually. JMeter uses memory to generate loads so the bigger the memory the bigger the virtual users number. We recommend you to use a Linux machine as its memory usage is better compared to Windows.
LoadRunner also provides the load generators additionally. There is a very handy UI that shows you the ramp up strategy according to your load generation configs. You can install the load generators in other machines. LoadRunner uses the computer resources better than JMeter. But mostly local installation won’t be the best solution to execute a performance test and you will need to move to the cloud to generate thousands of virtual users.
Reporting
Reporting is the part where LoadRunner is the most powerful. It has very handy reports where you can watch the execution and performance metrics instantly. It allows you to export them on fancy documents. On JMeter, this is very problematic as it’s reporting engine uses the computers memory inefficiently. But thanks to the amazing export functionality, you can export any numbers you collect during the execution in a CSV file. Then you can play with those numbers with any tool you want. Of course, this is not the ideal way of reporting, but you can combine the power of spreadsheets and get insight from the reports.
Both tools provide basic reports like hit per second, request time of a given request, latency, etc…
Open-Source & Community Support
Like all the other open-source tools, JMeter has a highly active community on the web. It has many contributors that keep adding plugins to the JMeter ecosystem. You can also find so many contributors on forums like stackoverflow. Most important thing is “if you have a question on JMeter, this question is most probably asked on stackoverflow” so it is so easy to find an answer.
As LoadRunner is owned by an enterprise company, it limits the contributors’ cause not everybody can acquire it. So less open source, less community help.
Stackoverflow gives a particularly good insight about this. In case we search JMeter and LoadRunner tags in it, JMeter tag has been used 14 times more compared to LoadRunner.
Integrations
It’s very easy to integrate the JMeter script with any build tool like Jenkins. Jenkins has special plug-ins to execute JMeter scripts. In case you use Loadium, we also provide a plugin (https://plugins.jenkins.io/loadium/) to execute JMeter scripts in our cloud. You can read the details on how to configure it on that page. (https://loadium.com/wiki/jenkins-integration/) By doing so, you can also take advantage of integrating your APM solution to your performance tests so you can monitor resources during your tests.
Same can be achieved for LoadRunner too. By using LoadRunner’s Jenkins plugin, you can get integrated into the LoadRunner Enterprise server and execute your test scripts.
CONCLUSION
In the end both tools are great but there are several reasons to decide which one to use. If you want to stick with SLA’s and get immediate help from a “real customer experience agent” then you will choose LoadRunner. But that will cost you a lot of money and time cause onboarding is not that easy. But with its smooth onboarding and its protocol support, JMeter is a great tool to try out. You can also use Loadium’s Trial plan to execute your performance scripts with 250 Virtual Users.
Enjoy Load Testing!
