Monday, March 1, 2010

QTP - Hybrid FrameWork - Part I

QTP - Hybrid Automation Frame Work


Initially I have presented this article at SlideShare


What is Automation Framework?

Framework is a wrapper around complex internal architecture which makes end user to interact with the system easily. It also define guidelines and set standards for all phases of ADLC (Automation Development Life Cycle).

This “Dual Function” (Hybrid) framework is a combination of three frameworks (Functional Decomposition, Data Driven, Keyword Driven) and (re-usability, advanced error logic, custom QTP methods, dual function) techniques.


Key features of Dual function Framework

1. Re-usability, low maintenance design (Dual Function).
2. Support different application environments (Development, Quality Control, production) and custom client settings.
3. Externally Configurable - Execute all the test cases, execute only failed test cases, execute test cases by test case id, execute test cases by using flags...can be configured in different ways.
4. Self Configurable and can run unattended with out results being overwritten.
5. Better ROI (Return Of Investment).
6. Real time status monitoring…display detail status during the test execution and automatically close the pop-up status in x sec.
7. Test status reporting in three ways…detailed, high level and summary log file.
8. System automatically send email notifications with test results once the test is complete.
9. Screen Capture on failed test step, notified in the test case itself, assign unique name and stored in the separate folder for easy accessing.
10. Automation test cases resemble manual test cases (Just replace with keywords and test data)
11. Easy way of creating test cases with test case generators (Select keywords from drop down).
12. Easy to maintain and develop scripts (DP, Custom Methods and Dual Function).
13. Test execution time stamp for each step, so that we can easily trace when the test was executed.
14. Same script can be executed on QTP 9.2, 9.5 and 10.0. Backward, forward compatibility.
15. If the test is running in different location, system automatically identify the place of execution using System IP. If the IP is unknown not in the defined list or changed, it would publish the IP address in the results.
16. If object is missing or object properties got changed on the application, system will notify object is missing and proceed with the next test case. This will not stop the test execution.
17. Calculate each page response time automatically.
18. Framework is designed in such a way that it can be initiated using AOM script or directly run using QTP GUI especially useful during debugging or script creation.
19. It would automatically stop automatically if specific number of test cases got failed in sequence, this will help us know there is serious problem in the application.
20. Multi language support.
21 . Sending test results as SMS.
22. It would automatically add new library files.
23. As the entire code reside in library files, total size of the framework is very small. This will help us to easily copy, backup and upload into version control tools.
24. Automatic results archiving facility, old results will not erased.


Advantages of Dual Function Framework and ROI (Return Of Investment)

1. Reduce testing time.
2. Improve testing productivity.
3. Improve product quality
4. Reduce QA costs.
5. Consistent test results.
6. Calculate page response time of all the pages.
7. Schedule test runs.



Frame Work Folder Structure



Framework contain different files associated with different functionality, all the files to be placed in the corresponding folders; so that it is easy to maintain.

Note: Framework can't identify the folders if folder names get renamed, except sub folders created under "AutomationTestDocuments".

AutomationTestDocuments folder contain sub folders relating to each test. If we have two tests, there will be two sub folders. Create new folders as per the test requirement. There is no restriction on the folder name.


Each test (ProductionLoginTesting from the above screen shot) contain 2 Files (Startup, Control) and 3 sub-folders (AutomationTestCases - contain test cases; ExecuationLog - system automatically create two new files once test is complete, based on the configuration, Test summary.txt and Response time.xls; ScreenShots - contain application screen shots, if there are any errors. It would automatically assign unique name with time stamp for each screen shot and notified in the test step itself.

Config folder contain xls files containing custom client settings.

ExecutedScripts folder contain copy of the test folder after execution. If we run the same test again and again (schedule on daily basis) results are over written. In order to preserver the test results, framework will copy the test folder into newly created  folder with time stamp, once the test got completed.



Frame_WorkGep folder contain QTP code of one action "KickOff" with single line of code "Call StartUP". Entire project code code reside in the VBS file.
Following are the reasons for choosing this kind of design.
1. VBS file are very light in terms of size.
2. Framework is entirely based on DP (Descriptive programming).
3. We can easily implement any configuration management tool.


Library folder contain application libraries which contain code relating to keyword functions and contain sub folder called "CoreLibraries" that contain files corresponding to actual framework only.


Dual Function Framework Architecture 




Note: High-level design intended to show in a simple manner. Many other scripts are called between intermediate stages.

QTP AOM Script
1. vbs file created using AOM.
2. Attached to windows scheduler.
3. Identify the place of execution using system IP. If script is executed on new location, it would consider IP address as place of execution.  
4. As there are many tests under "AutomationTestdocuments", script need to be updated with the test folder name and path that we are planning to execute.
5. Script will open the QTP and associate the test specified in the step 4.
6. Two parameters are passed as environment variables (location and test path).
7. Copy the results in "ExecutedScripts" folder to protect results from overwriting, if initiated through AOM script only.
8. Test can also be executed in standalone mode without AOM script, especially useful during test creation and debugging. 

KickOff Action
This is the only QTP action used in the framework, it contain "Call funKickOff" to initiate KickOff script.

KickOff Script
Contain global object declaration, initialization and controlling test execution mode. This script will initiate StartUp script.

StartUp script
This script will configure the framework as per the settings provided in the startup spreadsheet.

Control Script
Start up script will initiate control script. Based on the startup spreadsheet settings, script will read each test case from the control spread sheet and call the driver script with test case id, test case name. This script will also generate real time status of the test in the form of pop-up message and update the test status.

Driver script
Based on the test case id and test case name, it will map the test case lower bound and upper bond, then it will read each line of the test case and call corresponding key word functions. 

How lower bound is computed ?
Based on the test case id received from the control script, it will start verifying the line having the keyword "TestCaseID", if ID under P1 (Column F) is matching, then it is treated as lower bound.

How upper bound is computed ?
Once script identify the lower bound, it will start searching the immediate "STOP" key word, this will be considered as upper bound.

Then it will start reading all the keywords between lower and upper bound by calling corresponding key word functions and updating each step.




 

Application library
These functions will actually manipulate(enter data or perform certain action on the control) the application as per the design and send the status back to the driver script.

To understand different variables that I have created in the framework, look into the function available under this link.

Note: Above scripts contain complex validation and error logic techniques, just gave you high level overview. Following top down approach and each script is independent entity, this will help us to maintain framework with ease and change the functionality without effecting other scripts.


Start-up spreadsheet

Row 1 - Default URL - If you are using the same test in different environments (Development, Quality control, production), just change the URL, scripts would run without any issues. This will help us to run the tests in different environments with very little change.
Row 2 - DB Name - If you are accessing the database through QTP, you can specify the name after creating DSN.
Row 3 - Test Execute Flag - Drop Down values All,Yes,No,Empty - Execute Flag,Check,Test Case Ids
These settings are applied by the control script on the control spreadsheet.
All - Execute all the test cases
Yes(Y) - Execute test case with Y flag
No(N) -   Execute test case with N flag
Empty - Execute Flag - Execute test case with empty flag
Check - When selected, it would just generate test report with consolidate pass/fail status (if test is run multiple time, initially all test cases, next failed test cases) without actual execution the test. 
Test Case Ids - Execute only specific test case ids.
Row 4 - Test Execution Status Flag -  Drop Down values All,Pass,Fail,Bad Inputs,Empty - Execution Flag,Check,Test Case Ids
These settings are applied by the control script on the control spreadsheet.
If we want to control the test based on Pass/Fail status use this control.
All - Execute all the test cases
Pass - Execute test case with Pass status
Fail -   Execute test case with Fail status 
Bad Inputs -   Execute test case with Bad Inputs status 
Empty - Execute Flag - Execute test case with empty flag
Check - When selected, it would just generate test report with consolidate pass/fail status (if test is run multiple time, initially all test cases, next failed test cases) without actual execution the test.
Test Case Ids - Execute specific test case ids.
Row 3 and 4 can be used in different combination while executing the test, so that desired test cases are executed. Most of the combination's are considered.  
Row 5 - Execute Specific Test Case Ids - Mention test case ids that you want to execute, "Test Case Ids" flag need to be selected in Row 3 or 4. This will override all the settings.
Row 6 - Tester Name - Name of the tester who is executing the script, same name appear in control and log files after test execution.
Row 7 - Release no/ Module name.
Row 8 - Test Cycle - Mention test cycle.
Row 9 - Exceptions exist counter - Test would automatically stop once the failed test case count reach the counter value. It doesn't make sense to keep on executing the test with 50 or more failed test cases, some thing gone wrong badly.
Row 10 - Test status pop-up display - Drop down values On,Off,Valid Test Case
On - Always display real time test status pop-up.
Off - Don't display real time test status pop-up.
Valid Test Case - Display while executing valid test case, don't display for invalid test case, determined from the combination of Row 3 and 4 flag settings.
Row 11 - Test status pop-up display (sec) - How may seconds real time test pop-up need to be displayed, automatically close after specified seconds.
Row 17 - Object Highlighting - On/Off
On - It will highlight all the objects during test execution, reduce test execution speed.
Off - No highlighting
Row 18 - Silent Mode - On/Off
On -Framework generate many messages during execution, when test is getting executed unattended, we can suppress these message by selection On.
Off - User need to click OK for each framework message, used during debugging.
Row 19 - Calculate average page response time - On/Off
On - It will generate an excel file with each page response time and store it in the "ExecuationLog" folder. Extra coding required, need to modify excel COM model function as per the requirements.
Off - It will not generate any page response file.
Row 26 - Beep on any click - On/Off
On - Generate beep sound when system select any control. 
Off - No beep.
Reason for implementing this feature - One of the developer told me that you are not calculating page response time correctly there is flaw in page synchronization code. In order to prove that I am correct, I have implemented this feature. When system select any action on the control it will generate a beep, once page sync is complete, it will generate different beep. After showing the same to the developer, he agreed that my page response time code is correct.


Control Spreadsheet



Headers highlighted in YELLOW - User need to fill the columns.
Headers highlighted in WHITE - System automatically fill the columns during execution.
One control spread sheet for entire test.

Column 1 - Test Case Id - Each test case is assigned unique id in ascending order only.
Column 2 - Test Case Description - Optional, small description about the test case.
Column 3 - Execute Flag(Y/N/Empty) - Control script will retrieve the settings from the start up spread sheet Row 3 and check the flag. If both flags match, system will execute the test case else it will ignore it. This system will help us to quickly configure which test cases to be executed.
Column 4 - Test case file name - Need to specify the corresponding test case file name.  Each test can have multiple test cases if required, so that functionality can be distributed and easy to maintain.
Test case - Group of test steps(procedure) with specific test data to validate a requirement.
Test case file - Group of test cases placed in a file.
Column 5 - Response Time - Consolidated response time of all the test steps under specific test case.
Column 6 - Execution Status - Once test case id is executed, system will update the status with Pass/Fail/Bad Inputs (test data provided to the test steps are incorrect).
We can also use this flag to control test execution. Row 4 start up spreadsheet.
Column 7 - Execution Place or System IP / Status Message - Display execution place/ Error and check point messages, if there are any inconsistencies.
Column 8 - Time stamp - Test execution time stamp in Local time.
Column 9 - Test tester name - As specified in the start up spreadsheet.
Column 10 - Release no/ module name - As specified in the start up spreadsheet.

Test case Spreadsheet
Before Execution
 
Headers highlighted in YELLOW - User need to fill the columns.
Headers highlighted in WHITE - System automatically fill the columns during execution.

Test case spreadsheet consist of test steps similar to manual test cases.

Column A - Keyword - Perform specific action on the screen like enter data, retrieve data, check point. These keywords are selected from the drop down, not required to type the keywords. Once test case creator select the keyword, based on the design, column F to Q are highlighted and corresponding cells are inserted with comments, these are input parameters P1, P2, P3....P20. "LogInSignon" (Row 118) contain 4 parameters, P1 - User name, P2 - Password, P3 - Credentials are correct, P4 - Language.



After Execution
System start filling columns B, C, D and F during test execution.

Mapping of manual and automation test cases to check the coverage

 

We can create new column "Automation test case ID" in the manual test cases spreadsheet, so that there will be one to one mapping without missing any functionality.


Test summary log file





Generated and placed in the 'ExecutionLog" folder one test is completed.

It contain following information.
1. Place of execution.
2. Test start and end time with time zone.
3. Total executes test cases (Pass/fail/bad inputs) and ignored test cases (based on start up spreadsheet settings).
4. Star up spreadsheet details.
5. Information relating to sending email notifications.
6. Average page response time (depend on the start up spreadsheet settings)
7. Available PC physical memory before and after execution with time stamp.
8. Test termination message By system completed/ By user/ By exception exist counter.
9. Whether email sent successfully.
10. PC information.

It is a summary of Test results + Framework settings + PC info.




Real-Time status POP-UP message

 


It provide live health of your test, this can be configured in the start up spread sheet.
Above image is self explanatory.
If your test is running for 2 hours and don't have this feature, you will understand nothing if you look at your running test after one hour. It would be better to implement this feature as per your requirements, not so exhaustively as mentioned above. 

Note: Implementing these features will not have any performance bottleneck in terms of execution speed or memory usage.

 Test case generator (Keyword sheet to create test cases)
Simple Sheet
 

Test case generator spreadsheet contain two sheets
1. KeyList - Where you design keywords, same as above screen shot
2. Keyword - Where you create automation test cases by selecting the keywords from the drop down.
Macros are written to connect both sheets

How to create KeyWords?
Lets create keyword for the login screen, where you have User name, Password and Login button.
Flow - Enter user name, Password and select login button.
Keyword name - LoginSignOn 
Input Parameters - User name, Password
Assign two parameters P1, P2 under column E and F.
Create cell comments for P1 and P2 as user name and password.
O- Optional parameter
M - Mandate parameter (System will generate "BAD INPUTS" if users don't enter values and execute the script.

Points to be considered while creating Keywords
1. Standard CamelCase format.
2. Keyword granularity (never have same keyword doing same or overlapping functionality).
3. When creating complex keywords choose following format (product name + page Name + functionality)
4. Don't worry about the length of keyword, you are not typing it, just selecting from the drop down. It should be self explanatory.

Advantages of this design.
1. It is not required to enter test object properties in the test case, already implemented in the code.
2. We can created a document which contain screen shot of the page and corresponding keywords. By looking at it any one can create test cases.
3. Generally in keyword driven framework, we can enter one value or fire one method per line; in this design you can enter maximum of 20 values in a single line. 
4. It is not required to memorize all the inputs associated with a keyword, Just select the Keywork, it would automatically highlight the parameters and assign cell comments to each parameter, so that you will understand easily what P1, P2...parameters for.
5. Each key word is self explanatory designed using camel case, any person can understand it.


Complex Sheet with lengthy keywords

After creating the core framework, next job would be creating keywords as per the functionality and executing the test.





Continued... QTP - Hybrid FrameWork - Part II









---






33 comments:

  1. hi,
    I am fan of ur blog and it is quite informative.

    May I request for qtp script for testing 301 redirects of URLs ( 9.1 version being used)

    ReplyDelete
  2. Hello Phanee,

    I have not understood your post (301 redirects of URLs).

    ReplyDelete
  3. Amazing work.... I liked it a lot. Do you have any plans of sharing this framework i can suggest some architectural changes.I myself have worked on 6 different types of frameworks on QTP and QC.
    Mail me @ aditya_kalra@ymail.com

    Best Regards,
    Aditya Kalra.

    ReplyDelete
  4. Hello Ady,
    I have shared all the details with screen shots including framework architecture, still not sure what you are expecting. Let me know if you want any specific details. Your suggestions are welcome.

    ReplyDelete
  5. Hi Bharath,

    I'm sorry i should have been clear, i meant the code base that you are using in this framework. I shall be happy to share my best practices as well and not make it public...
    I have a small mailing list where i have lots of people who share common interests in automation.I shall be glad if you would like to join the same.

    Best Regards,
    Aditya

    ReplyDelete
  6. Hello Ady,
    Currently I will not be able to share the code.
    Framework contain lot of sensitive information like IP address, user names, passwords, client settings, port numbers... Isolating the information from 200 functions and making the code work with the new changes take time. I am currently dedicated to load testing, let you know once the isolation is completed. Mean while I am planning to publish in the blog, most of the methods used in this dual function framework.

    ReplyDelete
  7. Hey Bharath,

    Thats great nice to hear that. Will follow your blog religiously :)

    Best Regards,
    Aditya.

    ReplyDelete
  8. Hi Bharath,

    Bravo on such a detailed explanation of the framework. I've been reading articles about automation framework, but yours is the most practical and informative,

    Many Thanks,
    Jeevan

    ReplyDelete
  9. Hi Bharath,
    Good Work that is really good , We try this once ....

    ReplyDelete
  10. Hi ,Bharath Marrivada,
    I'm Luca. first at all nice blog.
    But i have a question:
    most of the Feature of the Automation framework (not the script but file structure report ecc..) it will be available from QC so i have a question :
    Why we need to implement a Framework like this ?
    Maybe i don't have understant the basic scope of the Automatic Framework

    Luca

    ReplyDelete
  11. Hello Luca,

    Good question.

    File structure and reports are available in QC, but I am generating reports with more details, can you compare my report with QC report and let me know the differences?
    If you take a car, engine provide raw power, that need to be harnessed through gear box and send it to the differential (tires). You need to have speedometer, fuel gauge, engine heat etc. Why do you need these things? To monitor and manage things efficiently. Sophisticated cars contain ABS, Traction control, cruise control, warning systems etc.

    In the same way QTP is like a car engine, it has the power to identify the objects that need to be harnessed through different conditions and error logic. Like a car, it also require real time status, log reports, email notifications, results as SMS etc to monitor and manage things efficiently.

    My scripts execute at three different geographical locations simultaneously, scheduled to run daily. It should not fail even if some objects are missing or site is down.

    Do you know how your car engine works? ABS works? In the same way QTP engineers not require to understand complete framework, just need to know how to design the scripts as per the framework standards (knowing how to drive the car).

    Executing test on BMW or Audi car framework, I mean having more features.

    Designing sophisticated framework is my passion, not only in QTP also in Loadrunner.

    Bharath.

    ReplyDelete
  12. Hi
    Bharath sir,

    i am big fan of u, why because Ur Dual Hy-bride Frame Work is Super,that key features and Scripting . Please Up Load the Folder Structure With Any Real-time Web Application like Mercury Tours. i am waiting for Ur Post and replay. thanks

    ReplyDelete
  13. Hi Bharath,

    Its quite an useful and informative post for Automation Framework. But as I am new to QTP and dont know much about frameworks so I would like to ask one question here:
    How and where to get Hybrid Framework from?

    ReplyDelete
  14. Hello Anuj,
    You need to design based on your project requirements, so that it can be used across your origination, I mean all the automation projects.

    ReplyDelete
  15. Hello Suresh,
    I am busy with other tasks, soon I will try to explain the architecture by using mercury tours.

    ReplyDelete
  16. Hi Bharath,

    Today I have seen your blog. I have gone through Dual function framework. It is superb,fantastic. I also became a big fan of you. I have 6+ yrs of experience in automation testing and I implemented almost 4 types of frameworks including keyword driven framework. Ofcourse i found keyword driven is the final ultimate. One issue I found with keyword driven is "Executing the script for 25 or more no of iterations as part of E2E testing for Pharma company"
    As per your example I can't pass all 25 or more than 25 data values from driver script. I.e where you are passing values from the file where you are maintaining Keyword list as well as Keyword. I think in real time it will be not possible. Am I right? If so, could you give me one of the best solution for this?

    Thanks,
    Sukur Babu.B

    ReplyDelete
  17. Hello Sukur,
    My hybrid framework can work for any number of iterations without any issues.

    I have restricted the number of data values that can be passed to a key work by 20, due to performance related issues (Looping through all the cells unnecessary) and it might confuse the user while typing the values. There are other techniques through which you can pass any number of values with out performance impact.

    -Bharath.

    ReplyDelete
  18. Thanks Bharath. And this sounds good.Is it possible to you to share those techniques.

    If possible, it will be great to me if you can explain me the following functions:
    1. garrBroPage((fungarrbropageFinRowIndex(1,1,1)+1,6))
    2. gsP(1)

    Is it ok to you to reach you on your mobile. If so, tell me your convenient time.

    ReplyDelete
  19. Hello Sukur,

    That explanation is not simple.
    gsP(1), gsP(2).... contain data values associated with the keyword. System will dump the values from the excel sheet on to this variables declared in array format.

    -Bharath.

    ReplyDelete
  20. Hi Bharath,

    OK. I understood. Anyway thank you very much for your quick replies.

    Sukur Babu.B

    ReplyDelete
  21. Hi Bharath,
    I am looking for this kind of framework for one of our Insurance client. Is it possible to share those excel workbook used for the keywordlist and other non sensitive material would be great helpful for my deeper understanding on this topic.
    Thank you very much for sharing your experiences

    Regards
    Satish

    ReplyDelete
  22. Hello Satish,

    I tried to explain it in a detailed way at high level, I hope this can help you in designing your own framework as per your requirements.

    Excel sheets consist of macros that will help end user for creating the error free test case and reading the contents from the sheet.

    But actual code and techniques used are complicated and not possible to explain here. If you require any professional help, you can contact me on my email provided in "View My Profile" link.

    ReplyDelete
  23. Hi Bharath,

    I like the framework very much but I could not calculate the lower and upper bound from excel sheet.

    Could you please help me by giving the code of how to get lower bound and upper bound in excel for test case id.

    Regards,
    Punit

    ReplyDelete
  24. Hello Punit,
    I will explain here with an example.
    Row1 Test case ID 1
    Row2 Step1
    Row3 Step2
    Row4 Step3
    Row5 Step4
    Row6 Step5
    Row7 STOP
    Row8 Test case ID 2
    Row9 Step1
    Row10 Step2
    Row11 Step3
    Row12 Step4
    Row13 Step5
    Row14 STOP

    Here you have two test cases, Test Case 1 and Test Case 2.
    To find the range of test case 1, Initially you need find the total number of rows (occupied cells, there is a function that give you the total count). Start reading from row1, check if there is a keyword (Test case ID or STOP). If there is a test case ID, check whether your test case ID (ID coming from the control file) match. If there is a match this row is lower bound(row 1), start reading till you reach STOP, this your upper bound(row 7). Now you got the range. This is simple logic, if you get it write now think, if there is no STOP indicator etc.

    ReplyDelete
  25. Hi Bharath,
    I am trying to implement some of the features in your framework in the one i am currently working on. one thing i can't figure out is how to share global variables between the scripts being called.
    are you using scriptinclude()? or are you using some other way of sharing them?

    ReplyDelete
  26. Hi Bharath,

    The information which u provided is very useful for the automation testers which they can implement in their projects and get the good results.Keep posting with latest updates on QTP frameworks...

    ReplyDelete
  27. Hi,

    I am totaly new to QTP, I have a requirement to design a QTP framework(Hybrid) with QC.
    As per my knowledge QC will help to manage the complete folder structure, and i need to design and place all my resources (Startup, driver script, object repository,data tables test scripts etc) in QC folder structure.
    After this i can start using QTP for script writting.
    Please guide me if I am going wrong.
    what exactly is required to do, in framework designing?? Can you suggest some material or pdf wchich i can refer for this activity.

    ReplyDelete
  28. Hi,

    Could you please share the source code of this framework?

    ReplyDelete
  29. Really its very nice blog to understand the Hybrid framework.

    ReplyDelete
  30. Hi Bharat,

    Thanks for such a nice information.

    Really its very nice way to provide the information about hybrid automation framework.

    ReplyDelete
  31. Hi Bharat,

    Can u also give us information about pure keyword driven framework?

    Thanks in advance.

    ReplyDelete
    Replies
    1. Any framework that encapsulate the object properties using key words, so that users can compose the test cases without knowing the underlying details.

      Delete
  32. Sir..Can you please share me the sample keyword driven pratical examples document my mail id is gopinath048@gmail.com

    ReplyDelete