NeoLoad Vs Loadrunner 9.50 (Ajax click & Script)

NeoLoad Vs Loadrunner (Ajax click & Script)

These days I have been working with NeoLoad, so I would like to show the comparison between NeoLoad and Loadrunner.

Link to NeoLoad

Link to Loadrunner

1. Size of the software.
Neoload - 140MB
Loadrunner - 2.2GB

2. Software installation.
NeoLoad - Quick, no dependency on other software and not require to restart the system.
Loadrunner - Need to install .Net, c++... before installing the actual software and need to restart the system multiple times.

3. Product developed in
NeoLoad - Java
Loadrunner - Microsoft Technologies and C

4. Supported OS
NeoLoad - Windows, Linux, Solaris
Loadrunner - Windows, Unix, lunix

5. Supported Platforms and technologies
Neoload:
Platform- .NET, J2EE
RIA - AJAX, FLEX, SilverLight, GWT, RTMP, Java Serialization,  Push Technologies
Web Services: SOAP
ERP: SAP and Oracle Forms
Integration Products: DynaTrace
Link to NeoLoad Technologies

Loadrunner:
It support even more technologies, except SilverLight, GWT, Java Serialization,  Push Technologies, dynaTrace. Following link contain complete details, HP sell the product in protocol bundle.

Loadrunner Protocol bundle list

View more documents from Bharath Marrivada.

6. Scripting Language
Neoload - Scriptless - All GUI driven, so less mistakes and provide facility to execute java script for additional functionality.
Attaching Screen shot.

Loadrunner - C Language

7. Script recording method
Neoload - It record each request and response by using a proxy server, similar to Fiddler.
Loadrunner - It records user actions at GUI level. Mercury could implement QTP technology in this protocol by using limited object properties.

8. Virtual user execution process
Neoload - Process each request (Parallel or sequentially), once request is completed after receiving the response, it doesn't render  and proceed with the next request. You can't access page DOM.
Loadrunner -  It is a hidden browser, it works similar to the real browser by executing the web page code, so correlation is not required . Naturally consume more CPU and memory. For more details follow this link

9. Virtual user CPU and Memory consumption
NeoLoad - Very less as there is no rendering and client script execution.
Loadrunner - High as there is rendering and client script execution.

10. Handling third party and custom components.
Neoload -  It can handle all the requests, but If the component send binary data, NeoLoad will only be able to play back what has been recorded unmodified. If this data contains session ID or other parameters requiring dynamic replacement, the test will not work.
Loadrunner - Not sure, I had lot of issues with FCK editor, UltraWebGrid, model pop-up...

10. Price
Neoload - Flat Price available on the web site (Select the Link), less than 1/3 of the loadrunner depending on the modules selected. Flex and dynaTrace are costly, remaining modules are less.
Loadrunner - It depends on the vendor, they sell in protocol bundle.

11. AJAX
Neoload - There is FORK action which can emulate asynchronous calls by using multi threading. I LOVE this functionality.
Loadrunner - There is no specific function to handle AJAX calls, they say it automatically handle, programmer has no control over it.

12. Handling dynamic content (Parametrization) 
NeoLoad -It automatically handle common dynamic content using Framework dynamic parameters, you can add and update existing content based on the project requirement.
You can also use variable extractor for extracting dynamic content by specifying
a. Left Boundary(Start String) and Right Boundary( End String)
b. Xpath expression.
b. Regular expression.
There is a facility to test the dynamic content on the page response to validate your expression, also we can copy and paste the expressions, very matured no scope for errors. Attaching the screen shot.

Loadrunner - There is automatic correlation(Recording settings) and manual correlation using
web_reg_save_param, there is no way to test your expression.


13. Access page DOM.
NeoLoad - No
Loadrunner - Yes

14. Creating scripts and scenario and analyzing reports.
Neolaod - Creating, executing and analyzing the reports are done using single GUI, all the files are stored as one project. Awesome design, it has removed headache of maintaining separate script, scenario and report files.
Loadrunner - You need to open separate applications for script creation, scenario creation and analysis. As time pass maintaining these files is a real challenge.

15. Comparison Report
NeoLoad - You can run two tests and easily find the differences by using comparison report.
Loadrunner - Need to perform comparison manually.

16. Server counters monitoring
NeoLoad - It is easy, not required to type any credentials for accessing, just add your windows user id under "Performance Monitor users" group on the server machine.
You can also set threshold limits, system will automatically generate alerts based on the settings.
Loadrunner - Need credentials for accessing the server counters. For sophisticated monitoring you may need to purchase HP-SiteScope. 

17. System defined variables
Neolaod - Variable change policy is good here, attaching the screen shot.

Loadrunner - Need to improve the variable change policy, attaching the screen shot.

18. GUI design in representing Request/Response
NeoLoad - For each request users can easily understand what are the POST and URL parameters and request, response is provided in TAB format. Attaching the screen shot.

Loadrunner - All the requests/response are bundled and put under "Generation Log"

19. Number of connections open simultaneously with the remote server per virtual user.
Neoload: Most browsers maintain two connections, we can configure to any number.
Loadrunner: Not sure.

20. Search and replace the content.
NeoLoad - Very sophisticated "Search and Replace" option, we can target specific content and perform the operation. Attaching the screen shots.

Loadrunner - Normal Search.

21. Validating server response.

How to validate server response ? (Select The Link)


NeoLoad - All the validations can be performed on one screen. Attaching the screen shot.

Loadrunner - Need to use code and GUI methods.



22. Flag or Mark Content
Neoload - If you are searching for a value and what to know in how may placed it is present? You can search for specific content, all these techniques are useful when dealing with dynamic content. You can perform the same operation while validating the script. Attaching screen shot.

Loadrunner - No such concept.


23. File upload Process
Neolaod - Just mention the file path in the post parameters, it is automatically copied to all the load generators during test execution.
Loadrunner - Mention the file path and manually copy the file in the specified location in all the load generators.


24. Load generators monitoring
Neoload - Automatically monitor CPU and Memory. It is always advisable to monitor load generators.
Loadrunner - Manually need to add the counters.



25. 32/64 Bit Load Generators
Neolaod - It has 32/64 Bit load generators. Better utilization of the hardware.
Loadrunner - Only 32 bit, it works on 64Bit machines, but in 32Bit mode.



26. Graph Template
NeoLoad - It is possible to create graph template, so that graphs will be arranged in the same way for all the results.
Loadrunner - Add graphs manually for each test.


27. Compare
NeoLoad - You can compare request, response with the actual recording and open the response in the browser if required. Attaching the screen shot.

Loadrunner - You can compare only the script, no option to compare request, response. Need to save the response in .html file and open in the browser if required.


28. Throughput
NeoLoad - Each request throughput is automatically captured, look at above screen shot.
Loadrunner - Need to use the function web_get_int_property( HTTP_INFO_DOWNLOAD_SIZE );

29. Security Testing
Neolaod - It can be used to perform HPP Attack.
Loadrunner - It can't be used because we can't manipulate request parameters..

30. File download process
NeoLoad - File is downloaded, but not stored in the load generators. It just record the total number of bytes downloaded, there is an option to insert a check point (Assertion).
Loadrunner - Actual file is downloaded on the load generators, this will help us to check the integrity of the file downloaded. But this is a complex process require extra coding.

31. Calculate page response time.
NeoLoad - It just capture the request, response time. It can't calculate page rendering time, you need to run QTP or Selenium test in parallel to capture the TRUE page response time.
How to capture page response time using QTP?
How to capture page response time using Selenium?
Loadrunner - It capture TRUE page response time which include request, response and page rendering time.

32. Creating transactions
NeoLoad - There is no facility to create start and end transactions to capture the response time. Each and every response time is automatically recorded using continues timer, need to group the requests into "containers" to capture the aggregate time. I felt NeoLoad need to improve here. I can't present the results directly, need to copy in an excel file in-order to present the results. Also there is no "Raw Data" provided, if we want to know the response time of each transaction. Just need to rely on the NeoLoad computation of average, 90%.
Loadrunner - Summary report contain all the transactions defined on the code that are easy to present in the report. There is a facility to download all the transaction response time in an excel format.

33. Parameterization
NeoLoad - All the server generated dynamic values need to me parametrized. There is an automatic parameterization at technology level(.Net, Java...) but this will not solve the problem completely, need to handle manually.
Loadrunner - No parameterization is required, loadrunner work like a real browser. Based on the requirements you many need to capture the response content and parameterize LR function. This is were loadrunner dominates other load testing tools. Same script can be successfully executed in any environment (Development, QC, Production having different URLs and test data), just by changing the URL at one place.
NeoLoad has answered this question by implementing the concepts of "Servers", where host name is captured in a variable that can be easily changed when required. In-order to implement this successfully, need to parameterize the content at very detailed level because each environment had different test data, so it had different dynamic values.

35. IP Spoofing
NeoLoad - Yes
Loadrunner - Yes

36. Specific Virtual user Stop/Restart
NeoLoad - No
Loadrunner - Yes

37. WAN Emulators
What are WAN emulators?
NeoLoad - There is no facility to add network effects, just limit bandwidth.
Loadrunner - It has integration with  SHUNRA Virtual Enterprise Suite to generate network effects.


38. Cloud Support
What is cloud load testing?
NeoLoad - Yes
Loadrunner - Yes.


39. JavaScript editor
NeoLoad - It has editor where we can compile the script for errors.
Loadrunner - No editor, need to write error free code. How to write JavaScript?


40. Share data between virtual users
NeoLoad - Shared queues 
Loadrunner - Virtual Table Server


41. Customer Support
NeoLoad - Excellent, very cooperative in answering all the questions. We could get 1 day - 500 users trial version to actually check the tool performance.
Loadrunner - I can't rate.

NeoLoad is one of the BEST tool available in the market for load-testing in terms of price, support and compatibility with web 2.0 . 

I would suggest following improvements for NeoLoad.

1. Option to Enable/Disable a specific request quickly, I think this will help during debugging.
2. Ability to monitor PASSED/FAILED transactions count separately (In Neoload Containers and assertions) during load test execution, these counters quickly asses the health of the test and analyze how many requests got processed, especially useful when requests execute in loop. We can get this count in test results after completing the test, I am more concerned during execution. For example

3. It is possible to generate load from cloud through partnership, ability to deploy Load generators on cloud using Amazon EC2 or GoGrid, NeoLoad will be running on our premises, only the load-generators deployed on the cloud.  
4. Currently tool is able to capture the response size, it would be better if it can display the request size.
5. It is possible to capture the page response time by grouping all the requests into a container, but it would be better by using Start and End transaction to capture the response time. If there are multiple scripts having same login procedure, I can define same transaction names in these two scripts, so that I can get aggregated results, instead of two separate login response time results. Also it would be great if the transaction values are dynamically updated during the test. 
6. While processing the dynamic parameters there is a progress bar which show system is busy identifying the content, but it is not showing how much % got completed. It would be better to show % completed (out of 500 requests, processed 100), this will be useful if the script size is big.
7. In some cases only, NeoLoad algorithm gets confused while recording HTTP 302 response, need to alter the code manually.
8. In some cases, While or Loop statement followed by If condition will not work. It works fine when we check the code through virtual user validity, it doesn't work when we run the same code in scenario. 
9. Ability to automatically add this variable "context.variableManager.setValue("computedVar",computedValue);" in the variable picker, so that we can get in auto-suggest.
10. Provision for sending messages to the NeoLoad console during test execution. If you are running the test with 100 users and what to know what Vusers are doing, you can send small messages to the console saying In Login Page, Updating, deleting, I logged in with 1test.com...
"lr_vuser_status_message" in Loadrunner. Attaching screenshot for more clarity.


11. Ability to provide raw values, list of the transactions response time and status that can be downloaded in excel format, so that we can compute the averages manually if required. Currently NoeLoad compute the average values and display it on the screen, what is the guarantee that it is calculating correctly or detailed invitational of each transaction. I think you people are building light weight controller, all these things make heavy.
12. Facility to display values in Output console(Small window to display values, just like log file but on the screen) during script validation, especially to check JavaScript variables content OR other values for debugging purpose. Similar to "Print" utility in HP Quick Test Professional. To over come this limitation I am using a dirty way.
13. I have not seen any option in NeoLoad where I can define maximum time a Request has to wait for an response.
14. Ability to Import/Export variable regular expressions from "Advance Parameters" window. You have provided the provision to add it to the framework that apply globally, but there are instances where I need to put this in few pages only. We can use copy method, but I need to create all those regular expression before copying to other required pages. I have seen instances where same dynamic value appear between different right and left boundaries on different pages, in this case I feel it would be useful.
15. There is no Minimum and Maximum values columns in the Runtime graphs.
16. Facility to Restart/Stop specific virtual user during test execution.
---