Performance Testing - On LAN and over the Internet(WAN).

Performance Testing - On LAN and over the Internet (WAN).

Generally we have performance test setup, where application servers and load injectors(generators) residing on the same LAN. We perform the tests and compare the response time reports with the SLA(Service Level Agreement) and declare test is PASSED.
When the same application is deployed on the data-center(Production Environment) and remote clients(real users) start accessing the application, they would notice difference in response time, that is not matching with the SLA. 

Why are clients accessing the same application observing high response time,which is running on similar hardware, that is TESTED & PASSED by the performance testing team ?

Why in many cases applications fail to perform to expected levels, despite load testing prior to roll-out?

We have been focusing on identifying software and hardware bottleneck, not considering the real world conditions like network impact, end user bandwidth etc.

There is difference between testing the application on LAN and over the internet.

Following factors effect the application response time when the request travel through internet.
1. Pack Loss - In internet terminology, every communication is sent as data packets.
A packet is a sequence of bytes and consists of a header followed by a body. The header describes the packet's destination and, optionally, the routers to use for forwarding until it arrives at its final destination.
Due to noise, some of the packets are lost or distorted. 
2. Packet Latency - All the packets will not travel the same path for a specific destination, internet nodes will select the shortest destination route based on number of factors,  if some of the packets are delayed, construction of the response would get delayed, in-turn response time.
Understand, how many hosts(routers) packet need to travel to reach destination server.
3. Packet Effects - Dynamic IP packet routing effects, including out-of-order packets, duplicated packets,
fragmentation phenomena, and TTL Effects (Time-to-live, decrement counter. Field in the packet header to count number of routers packet passed through) .
4. Link Faults - Possible damages to bit streams and possible disconnections.
5. Congestion - Sharp spikes in internet traffic, which may result in high latency or packet loss, or both.
6. Outgoing and Incoming Bandwidth - If your application throughput is not matching with the network, it would create congestion and increase response time.

Why is this effecting the server performance?

When data packets are lost, request  would be incomplete, so the application server has to resend the lost packets again with extra effort on the CPU. The stress on the servers increases, since more resources are needed to support remote end-users. Sessions are open longer, OS resources are occupied for more time and more concurrent threads are needed.

Without incorporating the above factors(WAN effects) into a load test, memory usage, thread usage, connection pool utilization, network stack, and other critical server resources can be significantly understated.


Select this link to understand internet line quality. 

Now, how to conduct TRUE performance test by taking above factors into consideration ?

Approach 1:  Install load generators at client location, where real users are accessing the application. This is a tedious process, clients network may choke during load test and security related issues my exist.

Approach 2: Perform test on your LAN by emulating the above factors. To create above factors locally, we need to use WAN Emulators.This can be achieved by using SHUNRA Virtual Enterprise Suite, it can record the above said factors from a remote user (or) real user and emulate the same in the LAN. Shunra provide plug-in software for Loadrunner. 

Approach 3: Generating load from cloud. What is cloud load testing? 
There are companies that can simulate load for any number of users from any part of the globe using cloud testing services (such as Amazon EC2)

For more information look into Cloud Load Testing.

Approach 3: Estimate impact of the network - Separately estimate the impact of the network on an application being deployed and manually factor that into result reports and deployment readiness findings.
This provides partial insight into the impact of the network, but totally ignores the interrelationships that network performance has on the application logic.Whenever the application logic or infrastructure configuration changes, there is an ongoing risk of encountering an unanticipated adverse impact and unhappy users.

Approach 4: Generate load from your premises in such a way that you are hitting the data center or production environment through internet. But purchasing or leasing hi-capacity internet lines just for conducting load test is very costly. If your internet connection capacity is less then the application throughput, your results will be incorrect.

Approach 5:  By using third party load generation services like Gomez  and Keynote. 


Gomez network exist in 168+ countries and 2,500+ ISP's. It consist of 500+ combinations of browsers and OS, 150+ commercial data centers, 5,000+ mobile supported devices and 1,50,000+ commercial grade desktops through which it is possible to generate enormous real user load. Internet giants like Google, Yahoo, FaceBook, LinkedIn... use these services. I feel it is ultimate in load testing for predicting true results. 

Attaching Gomez Network Screen shot.

Attaching Gomez Recorder screen shot, that records user actions on the page (similar to Loadrunner Click & Script protocol)

Attaching Gomez node selection (Load Generator) screen shot by Country|City|ISP.

Attaching Gomez results screen shot by City|ISP

Before analyzing results, always make sure that bandwidth is not the bottleneck. 



While performing tests over internet always compare internet bandwidth with the application throughput of all the virtual users. For example, if you are running test with 100 virtual users and consolidated throughput is 20Mbps, make sure your internet bandwidth is more than 20Mbps. If your application throughput and bandwidth don't match, there would be packet congestion, high response time is record, here it is not the problem with the application server but with the network.

While conducting performance test, response time should increase when more users are added, only when server CPU or other counters touch maximum limit, if this not the case, there would be some issue in the network or load generators.

Following posts are also related to performance testing.

Performance Testing Configuration or Setup
Performance related issues between browser and server
Performance Testing - TTFB, TTLB
Browser wars & End user performance, content display impact
Analyze Browser - JavaScript, AJAX, Rendering Details





---

Performance Testing - Validate server response.

Performance Testing - Validate server response.

For every request sent by client, server respond by sending the response.
In any performance testing tool following are the ways through which response can be validated.
In some tools these validations are performed by writing code, others by using GUI.

1. Check response status code. HTTP status code list.
2. Check response duration, whether the duration satisfy the SLA.
3. Check response size to make sure entire content is delivered by the server, measured in bytes (Greater than or equal to x bytes).
4. Verify response content. Content consist HTML, script...., not possible to verify the enter content. Place checkpoints on messages, id's, conformation text...



-----

Performance : Load : Stress Testing

Performance : Load : Stress Testing

Performance testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test with regard to speed, scalability and/or stability characteristics. It is also the superset of other classes of performance-related testing such as load and stress testing.
A load test is a performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes anticipated during production operations.
A stress test is a performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes beyond those anticipated during production operations. Stress tests may also include tests focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes while the product is subjected to other stressful conditions, such as limited memory, insufficient disk space or server failure.




-----

Loadrunner - Web(HTTP/HTML) Vs WEB,AJAX(Click & Script) protocol

Loadrunner - Difference between Web(HTTP/HTML) and WEB,AJAX(Click & Script) protocol in terms of virtual user

Web (HTTP/HTML) protocol script consist of series of requests written in C functions. Loadrunner start processing the requests one by one, once request is completed after receiving the response, it just forget what happened and proceed with the next request, so it consume less memory and don't process any web page code. Single web page consist of multiple requests and responses. Need to correlate the entire dynamic content,

Where as click & script protocol virtual user is a hidden browser like HtmlUnit, it works similar to the real browser by executing the web page code, so correlation is not required . By using very limited object properties, HP could implement QTP technology into this protocol. User can also access page DOM using java script similar to .Object in QTP. Naturally it consume more memory and create more load on load injectors. No correlation is required for the dynamic content, it is similar to the QTP scripts running in C style. If there are grid items and similar objects need to handle objects Html ID's.

Major issues with this protocol - Some GUI objects are not recognized, load generator hardware requirements, memory and CPU usage is unpredictable, when used with application having heavy AJAX and third party controls. 

If the application is using heavy AJAX and third party controls, it may consume lot of memory and CPU usage. Always monitor load generator memory and CPU usage, it should not go beyond 80%.
In these situation, we do have following issues when virtual users are running as "thread".
Some of the virtual users may fail with the following error message

Action.c(360): Error: C interpreter run time error: Action.c (360):  Error -- memory violation : Exception ACCESS_VIOLATION received.
Function Specific error
Action.c(360): Notify: CCI trace: Action.c(360): web_browser(0x02081b03 "Java_Script", 0x0208214f "ACTION", 0x02081292 "EvalJavaScript=igtbl_getRowById('ctl00xc...", 0x02082111 "LAST")

In-order to overcome above error, run the virtual user as "process" instead of "thread", now virtual users are stable and each is assigned separate memory.
Loadrunner create virtual user using mmdrv.exe process, if virtual users are running as process, virtual user count and mmdrv.exe process in the task manager count will be same.
Install load injectors with Windows 2003 64bit to accommodate more memory.

How to handle objects not identified by AJAX protocol.





---

Test Broadband or Internet speed and line quality.

Test Broadband or Internet speed and line quality.

Every one would be curious in understanding speed and quality of the internet that you are using.

Speed of the connection is measured in Upload and Download speed.
First we will understand unit of measurement for internet speed.

1 Mbps(megabits per second) = 1000 Kbps(kilobits per second) = 125 KBps(kilobytes per second)

Always pay attention to (b)Bits and (B)Bytes.
Generally service providers(Airtel, Tata Communications, Youtele) measure the speed in  Mbps or Kbps.
When you are downloading the file from the internet, most of the browsers measure the speed in KBps.
Attaching screen shot, where data is getting downloaded at 143 KBps, downloading at the speed of 1.144 Mbps.

Select this link to understand internet speed unit conversions.

Speed over the internet vary greatly from one location to another. Speed of contacting a server in Washington DC and a server in Tokyo would be different, so while calculating the speed, you need to select the nearest server and perform the speed test in order to get accurate results.

Speed Test Mini - Quick way of checking the internet speed. 

Speed Test Mini from www.WhatIsMyIP.com

I have used speedtest.net to calculate the speed, which has the facility to select the servers across the globe. It is free, you can test your connection now. Select "Begin Test" OR select a server in the globe and  select "Begin Test"

You can directly select the location from the screen or if you want to run the test repeatedly, under setting link, make sure you have selected correct "Preferred Server" and "Unit of speed measurement".
Attaching screen shot of the results.

There is "World Results" link on the top of the page that will provide global statics.
These reports will help you understand which internet service provider (ISP) is providing highest band width in your city (I have selected Hyderabad). Attaching the screen shot.

Line Quality can be measured using following items.


Ping - Amount of time taken by packet of data to travel from a computer to another computer and back. Time is measured in Milli seconds, less then 100 is acceptable.

Packet Loss - In internet terminology every communication is sent in terms of packets. Due to noise some packets are lost.

Packet Latency -Delay while receiving the data packets at receiver end.

Jitter - Time difference(variance) while measuring successive ping tests. Zero is good,

I have used pingtest.net to calculate the line quality, it is free you can test your connection now.

You can directly select the location from the screen or if you want to run the test repeatedly, under setting link, make sure you have selected correct "Preferred Server".
Attaching screen shot of the results.

Above concepts are useful in conducting application performance test over the internet, I will explain those in future posts.


---