Gomez Vs NeoLoad Vs Loadrunner Vs Selenium
One of my friend asked me for the comparison between Gomez and NeoLoad. I have included Loadrunner Ajax(Click&Script) protocol, as it is a well known performance testing tool. There is separate post for NeoLoad Vs Loadrunner. Also included Selenium function testing tool in the comparison because it can be used as performance testing tool and Gomez use Selenium scripts to run functional tests. HP QTP can't run parallel and Cross-Browser tests, so it is not included in the comparison.
Before starting the comparison, lets understand the performance issues between Server and Browser with few images, so that you can choose the tool in a better way.
Things can go wrong in any of the below mentioned places, Is your tool capable of identifying it.
Below are the issues at the application level. Below screen shot is taken from dynaTrace, excellent core application performance monitoring tool.
Currently there is no single browser dominating the web, every browser has its own share. You have full control of your data center, what about data coming from the third party sites. Today's websites are complex amalgamation of own and third party services. On a average browser need to resolve 8.3 different hosts (Different IP address) to display the content. Attaching the screen shot for more clarity.
Why will this impact the performance? Every browser has its own JavaScript engine, so the page rendering happen differently and have different parallel downloads. Website that work for one browser may not work properly with other browser at the some locations due to issues in third party services, If you can't detect it, you will loose the business. For more understanding select this link. Read this post to understand the rendering differences between FF and IE.
Small Description
Gomez is a portal where user can conduct Performance tests(Any browser/OS, Mobile devices), Cross Browser tests, monitor networks and functional tests without any hardware requirements. Entire execution happen at Gomez servers.
NeoLoad is a pure performance testing tool that record and replay HTTP(S) requests.
HP Loadrunner is a pure performance testing tool which support different protocols.
Selenium is an open source tool that is designed to conduct functional testing, but by using Selenium GRID we can run parallel tests in different browser environments.
Licence Cost
Gomez - SaaS model.
NeoLoad - Number of virtual users, technology and monitoring services.
Loadrunner - Number of Virtual users and Protocol bundles.
Selenium - Free
About The Software
Gomez - It has Gomez screen recorder(GSR), once you record the script and make necessary dynamic value changes, you can upload into GPN(Gomez Performance Network) portal. You can access this site online and conduct tests, view results.
NeoLoad - Around 160MB software that is written in Java need to be installed on your workstations. If you are generating more load, need to install separate LoadGenerator software on separate server. Also LoadGenerators can be deployed from the cloud.
Loadrunner - Around 2.5GB software that has separate applications for recording, test execution and analysis. Also has cloud support.
Selenium - Need to have Java and set of Selenium libraries that can be executed using any language that support HTTP library (Java/.Net/Python/Ruby...)
Script Recorder
Gomez - All the user clicks or actions are recorded, once user select an item on the screen, system automatically generate JavaScript associated with that click (document.GetElementByID("LoginBtn")), use Firefox engine to record the script. There are different actions available Navigate, Type, Wait, Form Fill....... It can identify the screen objects using DOM or CSS. There are different built in Gomez functions that can provide data and reference to different elements. You have got the option to use User clicks OR directly POST the form. During replay, user action associate JavaScript get executed in the Gomez injected JavaScript core inside the browser.
NeoLoad - No user clicks are recorded, It record Request/Response during form submit. During replay, it re-executes the requests that re recorded after changing the dynamic values.
Loadrunner - Ajax(Click&Script) protocol is suitable for WEB2.0 applications, this protocol has HP QTP technology with limited object properties. User clicks are recorded as C functions.
web_click("ctl00$RightContent$ddlCycleNo",
"Snapshot=t14.inf",
DESCRIPTION,
"Name=ctl00$RightContent$ddlCycleNo",
"id=ctl00_RightContent_ddlCycleNo",
"FrameName=contentFRAME",
ACTION,
"Select=B",
LAST);
It records only in IE, I think these c functions are converted into JavaScript to execute on the Loadrunner browser.
The same functions are executed during replay in Loadrunner proprietary browser.
Selenium - It has Selenium IDE that record and replay script as Firefox add-on. Also you can directly write script in any of the following languages(Java/.Net/Python/Ruby...)
Selenium - For sample code select this link.
Script Replay
Gomez - It replay the script in real and Gomez browser so there is page rendering.
Neoload - No browser, just replay of HTTP(S) requests.
Loadrunner - It replay the script in Loadrunner proprietary browser so there is page rendering. Lot of issues when page contain other objects.
Selenium - It replay the script in real browser so there is page rendering.
LoadGenerators
Gomez - It network exist in 168+ countries and 2,500+ ISP's. It consist of 500+ combinations of browsers and OS, 150+ commercial data centers, 5,000+ mobile supported devices and 1,50,000+ commercial grade desktops through which it is possible to generate enormous real user load.
Backbone are servers deployed on cloud to generate massive load.
Last Mile systems acts like a real users, executing single test by residing on real network, devise(Browser OR Mobile) and location. With massive last mile systems deployed around the word, it would be difficult for any one to challenge Gomez Network
NeoLoad - Deployed in the premises or on the cloud. Load getting generated from few places only.
Loadrunner - Same as above.
Selenium - There is no concept of Loadgenerators in selenium, using GRID we can run multiple browsers on different systems.
WAN Emulators
What are WAN emulators?
Gomez - It is not required, as tests are getting executed on real network, device and location.
NeoLoad - There is no facility to add network effects, just limit bandwidth.
Loadrunner - It has integration with SHUNRA Virtual Enterprise Suite to generate network effects.
Selenium - There is no facility to add network effects.
Multi-Browser support
Gomez - It support Firefox, IE and Gomez proprietary agent. Also it consist of 500+ combinations of browsers and OS...Amazing, with "Reality View XF" where screen shots of different browsers are shown in a single page and compared against selected screen shot to find object devastation.
NeoLoad - It doesn't execute in a browser.
Loadrunner - No, it execute in Loadrunner propriety browser.
Selenium- Yes, you need to have own software and hardware. If you don't have consider SauceLabs.
Test execution length
Gomez - You can schedule your test and run for any amount of time. You have also got facility to stop collecting metrics during server maintenance, so that averages are not deviated. All the results are stored in the GPN.
NeoLoad - If you run the test for 4 days, I feel system would get stuck while processing the metrics.
Loadrunner - I think the same as NeoLoad
Selenium - You can schedule using Hudson-CI.
Response time and Test results
Gomez - It provide results in great detail. You can get based on Geography, ISP, Location, Browser type and even browser rendering details(Object level). Each browser JavaScript engine renders in different way, this will be helpful us understand the cross browser performance impacts at end user. It also has the capability to take the browser screen shot during error.
NeoLoad - Container and request level.
Loadrunner - Record at transaction level.
Selenium - There is no built in facility to calculate response time, need to consider other methods.
Functional Testing
Gomez - Can execute scripts created using Selenium IDE.
NeoLoad - No
Loadrunner - No
Selenium - It is a function testing tool with cross-browser and parallel test execution capability.
Mobile Platform Support
Gomez - Yes
NeoLoad - No
Loadrunner - No
Selenium - Yes
Resource Monitoring
Gomez - It monitors entire delivery chain, from server, ISP to the end user.
NeoLoad - Monitor the server resources and throughput.
Loadrunner - Same as above.
Selenium - No resource monitoring capability.
Flex Support
Gomez - Support Flex/Silver-light, I am not sure how they are able to select Flex objects. As Flex is proprietary tool doesn't expose the methods in the DOM, generally .swf file need to be compiled with extra libraries. This compilation happen on fly while data getting downloaded into the browser, but increases number of objects and response time.
NeoLoad - It convert Binay data into readable format and then convert back after changing dynamic values as it is recording at request/response level, which is different from other reorders.
Loadrunner - It support, But I had issues identifying the objects.
Selenium - Yes, Select this for more information.
Presentations and Whitepapers
Gomez - Awesome, I love to read those presentations.
NeoLoad - Only pertaining to the tool.
Loadrunner - Only pertaining to the tool.
Selenium - As it is a open source tool, some time getting proper information would be difficult.
How to Choose and Use
Gomez - If your application is being used across different location or geographies with different browser combinations and your application has CDN or other third party services. Gomez is the best solution. It is ultimate in performance testing.
NeoLoad - If want to test the application inside the firewall OR if your application is used in specific places, generate the load from Cloud. For calculating true response time, execute Selenium test for each location with required browser combinations. It is not easy to generate enormous load using real browsers as they require lot of resources, the best combination would be generating part of the load using HTTP(S) request servers as they are very cheap and the remaining part from real browser combinations.
Loadrunner - Same as Neolaod, you can use HP QTP to calculate response time, If there are more browser combinations need to move to Selenium.
Selenium - User Selenium GRID or other vendors to generate load. Some companies link BrowserMob, PushToTest use Selenium GRID to conduct performance testing.
KeyNote Systems is similar to Gomez.
---
One of my friend asked me for the comparison between Gomez and NeoLoad. I have included Loadrunner Ajax(Click&Script) protocol, as it is a well known performance testing tool. There is separate post for NeoLoad Vs Loadrunner. Also included Selenium function testing tool in the comparison because it can be used as performance testing tool and Gomez use Selenium scripts to run functional tests. HP QTP can't run parallel and Cross-Browser tests, so it is not included in the comparison.
Before starting the comparison, lets understand the performance issues between Server and Browser with few images, so that you can choose the tool in a better way.
Things can go wrong in any of the below mentioned places, Is your tool capable of identifying it.
Below are the issues at the application level. Below screen shot is taken from dynaTrace, excellent core application performance monitoring tool.
Currently there is no single browser dominating the web, every browser has its own share. You have full control of your data center, what about data coming from the third party sites. Today's websites are complex amalgamation of own and third party services. On a average browser need to resolve 8.3 different hosts (Different IP address) to display the content. Attaching the screen shot for more clarity.
Why will this impact the performance? Every browser has its own JavaScript engine, so the page rendering happen differently and have different parallel downloads. Website that work for one browser may not work properly with other browser at the some locations due to issues in third party services, If you can't detect it, you will loose the business. For more understanding select this link. Read this post to understand the rendering differences between FF and IE.
Small Description
Gomez is a portal where user can conduct Performance tests(Any browser/OS, Mobile devices), Cross Browser tests, monitor networks and functional tests without any hardware requirements. Entire execution happen at Gomez servers.
NeoLoad is a pure performance testing tool that record and replay HTTP(S) requests.
HP Loadrunner is a pure performance testing tool which support different protocols.
Selenium is an open source tool that is designed to conduct functional testing, but by using Selenium GRID we can run parallel tests in different browser environments.
Licence Cost
Gomez - SaaS model.
NeoLoad - Number of virtual users, technology and monitoring services.
Loadrunner - Number of Virtual users and Protocol bundles.
Selenium - Free
About The Software
Gomez - It has Gomez screen recorder(GSR), once you record the script and make necessary dynamic value changes, you can upload into GPN(Gomez Performance Network) portal. You can access this site online and conduct tests, view results.
NeoLoad - Around 160MB software that is written in Java need to be installed on your workstations. If you are generating more load, need to install separate LoadGenerator software on separate server. Also LoadGenerators can be deployed from the cloud.
Loadrunner - Around 2.5GB software that has separate applications for recording, test execution and analysis. Also has cloud support.
Selenium - Need to have Java and set of Selenium libraries that can be executed using any language that support HTTP library (Java/.Net/Python/Ruby...)
Script Recorder
Gomez - All the user clicks or actions are recorded, once user select an item on the screen, system automatically generate JavaScript associated with that click (document.GetElementByID("LoginBtn")), use Firefox engine to record the script. There are different actions available Navigate, Type, Wait, Form Fill....... It can identify the screen objects using DOM or CSS. There are different built in Gomez functions that can provide data and reference to different elements. You have got the option to use User clicks OR directly POST the form. During replay, user action associate JavaScript get executed in the Gomez injected JavaScript core inside the browser.
NeoLoad - No user clicks are recorded, It record Request/Response during form submit. During replay, it re-executes the requests that re recorded after changing the dynamic values.
Loadrunner - Ajax(Click&Script) protocol is suitable for WEB2.0 applications, this protocol has HP QTP technology with limited object properties. User clicks are recorded as C functions.
web_click("ctl00$RightContent$ddlCycleNo",
"Snapshot=t14.inf",
DESCRIPTION,
"Name=ctl00$RightContent$ddlCycleNo",
"id=ctl00_RightContent_ddlCycleNo",
"FrameName=contentFRAME",
ACTION,
"Select=B",
LAST);
It records only in IE, I think these c functions are converted into JavaScript to execute on the Loadrunner browser.
The same functions are executed during replay in Loadrunner proprietary browser.
Selenium - It has Selenium IDE that record and replay script as Firefox add-on. Also you can directly write script in any of the following languages(Java/.Net/Python/Ruby...)
Selenium - For sample code select this link.
Script Replay
Gomez - It replay the script in real and Gomez browser so there is page rendering.
Neoload - No browser, just replay of HTTP(S) requests.
Loadrunner - It replay the script in Loadrunner proprietary browser so there is page rendering. Lot of issues when page contain other objects.
Selenium - It replay the script in real browser so there is page rendering.
LoadGenerators
Gomez - It network exist in 168+ countries and 2,500+ ISP's. It consist of 500+ combinations of browsers and OS, 150+ commercial data centers, 5,000+ mobile supported devices and 1,50,000+ commercial grade desktops through which it is possible to generate enormous real user load.
Backbone are servers deployed on cloud to generate massive load.
Last Mile systems acts like a real users, executing single test by residing on real network, devise(Browser OR Mobile) and location. With massive last mile systems deployed around the word, it would be difficult for any one to challenge Gomez Network
NeoLoad - Deployed in the premises or on the cloud. Load getting generated from few places only.
Loadrunner - Same as above.
Selenium - There is no concept of Loadgenerators in selenium, using GRID we can run multiple browsers on different systems.
WAN Emulators
What are WAN emulators?
Gomez - It is not required, as tests are getting executed on real network, device and location.
NeoLoad - There is no facility to add network effects, just limit bandwidth.
Loadrunner - It has integration with SHUNRA Virtual Enterprise Suite to generate network effects.
Selenium - There is no facility to add network effects.
Multi-Browser support
Gomez - It support Firefox, IE and Gomez proprietary agent. Also it consist of 500+ combinations of browsers and OS...Amazing, with "Reality View XF" where screen shots of different browsers are shown in a single page and compared against selected screen shot to find object devastation.
NeoLoad - It doesn't execute in a browser.
Loadrunner - No, it execute in Loadrunner propriety browser.
Selenium- Yes, you need to have own software and hardware. If you don't have consider SauceLabs.
Test execution length
Gomez - You can schedule your test and run for any amount of time. You have also got facility to stop collecting metrics during server maintenance, so that averages are not deviated. All the results are stored in the GPN.
NeoLoad - If you run the test for 4 days, I feel system would get stuck while processing the metrics.
Loadrunner - I think the same as NeoLoad
Selenium - You can schedule using Hudson-CI.
Response time and Test results
Gomez - It provide results in great detail. You can get based on Geography, ISP, Location, Browser type and even browser rendering details(Object level). Each browser JavaScript engine renders in different way, this will be helpful us understand the cross browser performance impacts at end user. It also has the capability to take the browser screen shot during error.
NeoLoad - Container and request level.
Loadrunner - Record at transaction level.
Selenium - There is no built in facility to calculate response time, need to consider other methods.
Functional Testing
Gomez - Can execute scripts created using Selenium IDE.
NeoLoad - No
Loadrunner - No
Selenium - It is a function testing tool with cross-browser and parallel test execution capability.
Mobile Platform Support
Gomez - Yes
NeoLoad - No
Loadrunner - No
Selenium - Yes
Resource Monitoring
Gomez - It monitors entire delivery chain, from server, ISP to the end user.
NeoLoad - Monitor the server resources and throughput.
Loadrunner - Same as above.
Selenium - No resource monitoring capability.
Flex Support
Gomez - Support Flex/Silver-light, I am not sure how they are able to select Flex objects. As Flex is proprietary tool doesn't expose the methods in the DOM, generally .swf file need to be compiled with extra libraries. This compilation happen on fly while data getting downloaded into the browser, but increases number of objects and response time.
NeoLoad - It convert Binay data into readable format and then convert back after changing dynamic values as it is recording at request/response level, which is different from other reorders.
Loadrunner - It support, But I had issues identifying the objects.
Selenium - Yes, Select this for more information.
Presentations and Whitepapers
Gomez - Awesome, I love to read those presentations.
NeoLoad - Only pertaining to the tool.
Loadrunner - Only pertaining to the tool.
Selenium - As it is a open source tool, some time getting proper information would be difficult.
How to Choose and Use
Gomez - If your application is being used across different location or geographies with different browser combinations and your application has CDN or other third party services. Gomez is the best solution. It is ultimate in performance testing.
NeoLoad - If want to test the application inside the firewall OR if your application is used in specific places, generate the load from Cloud. For calculating true response time, execute Selenium test for each location with required browser combinations. It is not easy to generate enormous load using real browsers as they require lot of resources, the best combination would be generating part of the load using HTTP(S) request servers as they are very cheap and the remaining part from real browser combinations.
Loadrunner - Same as Neolaod, you can use HP QTP to calculate response time, If there are more browser combinations need to move to Selenium.
Selenium - User Selenium GRID or other vendors to generate load. Some companies link BrowserMob, PushToTest use Selenium GRID to conduct performance testing.
KeyNote Systems is similar to Gomez.
---
Posts
Thanks for giving the detailed info abt Gomez Vs NeoLoad Vs Loadrunner Vs Selenium.
ReplyDeleteKeep up your good work!
http://qualitypointtech.net/ebook/index.php
very nice and good overview.
ReplyDeletethanks!
http://cgsjava.blogspot.com/
This comment has been removed by the author.
ReplyDeleteHi Bharath,
ReplyDeleteVery good explanation of the difference between tools. Can you also help me with an explanation similar to the abbove but between Gomez and Keynote. If i want to select between these 2 which would you prefer.
Thanks & regards,
Bindu
hi
ReplyDeleteI'm new testing what wants to do performance testing .So could you pls suggest which tool will be good -wapt pro or loadrunnner
Hello Nitish,
ReplyDeleteIt depends on the complexity of your web application, user location and your project budget.
There are many tools available
NeoLoad, Loadrunner, WebLoad, JMeter...
Hello Marrivada
ReplyDeleteI would like to perform a Outlook webaccess access (OWA) test to simulate 15000 concurent users,
Do you have any advices or experiences on this kind of test.
The web site is reachable from Internet.
Best regards
David
Hello David,
ReplyDeleteI have not done any test on OWA.
Just I gave a try with my OWA credentials using Neoload, I was able to record and replay the script.
So, WOuld it be safe to say...
ReplyDeleteWhile your product is in Pre-delivery mode Application development, Pre-release, BEHIND the firewall, that LoadRunner makes the most sense, but once it is pushed out to the world, Gomez is best at telling you how it is behaving in the eyes of real users?
Thanks for sharing the post. This post have all the flowcharts regarding the loadrunner services which are really easy to understand. Thanks for sharing the post. Really informative post shared..
ReplyDeleteLoadrunner
Hi Bharath, congrats for this great article pointing out the challenges of load testing and the actual strengths and weaknesses of the popular load testing tools.
ReplyDeleteAs an engineer in Neotys, software editor of NeoLoad, I would like to give you a few information that needs to be updated in your article:
About network emulation, NeoLoad is far beyond the other bandwidth emulation tools.
In our new release (4.1), NeoLoad can reproduce various network conditions by limiting bandwidth and simulating latency and packet loss.
For more information about network emulation, see http://www.neotys.com/product/network-emulation.html
About browser emulation, NeoLoad can emulate various browsers. Not only regular IE, Chrome, or Firefox browsers, but also mobile ones like Safari on iPad.
For each of them, more than the Agent field, NeoLoad generates HTTP requests at the same pace (e.g. number of open connections) as real browsers would do.
At last, about mobile platform support, on top of the functions described before, we have an exclusive recording system — see ʺRecording Mobile Devices in Tunnel Modeʺ: http://www.neotys.com/product/mobile-load-testing.html — based on listening to the communication between the real mobile devices and the servers, with a DNS hack that NeoLoad manages.
NeoLoad can record apps from both Mobile emulators and real mobile
devices, allowing you to choose the more convenient solution depending
on the environment.
I hope this will help keep your great article up to date. Also, I would be pleased to get your feedback on these news features.
Hervé Servy, Neotys Performance Engineer
Thank you Herve Servy for the update.
DeleteI have used Neoload for 2 years, really good product.
NeoLoad is a versatile tool. Some of the aspects like learning curve, zero scripting, correlation are not taken into account in this article. Also the complexity of the applications supported (Oracle Forms/ SIEBEL etc ) by NeoLoad and pricing gives it great competitive edge. The support is excellent during evaluation itself. I do not work for NeoLoad. I did an evaluation for NeoLoad for our company. Also regarding mobile support it is already incorporated. Please correct me if I am wrong.
ReplyDelete