A Guide to Automation Frameworks

A test automation framework is a comprehensive set of guidelines used to produce beneficial results of the automated testing activity. These guidelines may include:

• Common practices
• Assumptions for the desired outcome
• Test tools (software) and interfaces
• Test libraries
• Coding standards

In more technical terms, author Ashish Bhargava writes that a framework “facilitates a standard way for modifying, adding, and deleting the [test] scripts and functions,” and provides “scalability and reliability with less effort” (Designing and Implementing Test Automation Frameworks with QTP, 2013).

In less technical terms, consider a cooking example. You are creative, intuitive, and equipped with skills to operate kitchen tools. By following a recipe with step-by-step information, cooking methods, and results to measure your outcome against, you can exceed the results you would get without the recipe. Combining the essential tools of cooking with the guidelines provided by the recipe, you maximize your results.

Think of a test automation framework the same way you think of cooking with a recipe. You could prepare the food without observing any of the guidelines in the recipe, just as you could automate testing activities without a framework in place. The question becomes, how reliable are your methods? Can you recreate the results of your cooking efforts as quickly as possible? An automation framework, like a recipe, provides testers with guidelines to leverage reusable automated tests rapidly and reliably to achieve specific testing goals.

The titans of information technology like Google, Amazon, and Microsoft develop software at a rapid pace to meet the demand for their products and services. For these companies—and the IT industry at large—to reap the benefits of test automation, they rely on automation frameworks to guide their efforts.

Experts from the test community refer to some key elements of effective test automation frameworks. These frameworks should:

• Be scalable to future requirements
• Be easy to maintain and reusable
• Be reliable and consistent
• Maximize test coverage
• Provide reporting of automated test results
• Improve efficiency of automated test activity

These elements offer clues to understanding why test automation frameworks are critical to today’s modern software development methods. Test automation is a fundamental part of Agile. Various core practices of Agile, such as Continuous Integration (CI), Continuous Delivery, Test-Driven Development (TDD), and Behavior-Driven Development (BDD) rely on the efficiency and reliability of test automation. The automation frameworks provide the foundation that supports a variety of the automated software tests including:

• Unit testing
• Functional requirement testing
• Performance testing
• GUI (graphical user interface) testing
• Regression testing
• Integration testing
• System load testing
• Security testing

Bas Dijkstra is an independent test automation professional and author of various articles on test automation topics. He has more than ten years of experience designing and developing test automation and service virtualization solutions. Dijkstra prefers the term solutions to frameworks when discussing the topic.

“I’m referring to all the code, tools, and libraries that support the actual test automation scripts to be executed,” says Dijkstra. According to Dijkstra, the code, tools, and libraries have different tasks including (but not limited to):
 

• Acting as a test runner
• Reporting
• Test data generation and management
• Synchronization and error handling
• Abstraction

Here is a quick summary of the standard types of test automation frameworks in use. If you prefer to learn more about each framework in detail, the sections later on in this article provide more depth, including some pros and cons of each. Use the navigation on the left sidebar of this article to jump to detailed sections on each type of framework.

• Linear Scripting Framework: Recording and replaying test scripts in sequential (“linear”) fashion with little or no modification.
• Data-driven Framework: A constant source of test criteria (internal or external data) specifies the test scripts to run.
• Keyword-driven Framework: Tables on a spreadsheet determine the action of a test script based on a library of functions for an assigned keyword.
• Modular Testing Framework: Modules of an application under test are divided and tested with individual test scripts that can be combined to make larger test scripts.  
• Hybrid Testing Framework: A combination of frameworks to leverage the strengths of each.

These terms and descriptions are not necessarily critical to the decision of which framework you apply to test automation projects. Dijkstra avoids focusing on the frameworks with clients and teams to prevent distraction from what’s important when designing and implementing automation solutions. “I much prefer to talk about the goals that these frameworks are trying to achieve,” says Dijkstra.

Having the guidelines in place to realize the benefits of automated tests is the main objective of a test automation framework.

• Reusable Tests: Properly designed test automation frameworks deliver routinely stable automated tests that are easier to maintain and reusable.
• Lower Cost of Maintenance: Reusable automated tests have a lower cost of maintenance. More maintainable and reusable automated tests help meet budget demands and achieve the rapid software release schedules of modern software development.
• No Manual Intervention: There are exceptions where temporary manual intervention is acceptable, but according to Dijkstra, this should not be the norm when you have the appropriate framework in place. “Always work towards unattended. Even if you’re only running your automated tests once a month, they should be able to run unattended,” he says. “Otherwise, you’ll have a hard time later on when you try to speed up the development and delivery process (to Continuous Delivery), but your automated tests do not support Continuous Testing.”
• Transparent Reporting: Dijkstra uses the term right reporting, which he describes as clear, unambiguous, and targeted reporting towards the right audience. “This can be a manager, a developer, or an automated build tool,” he adds. “Each of which requires a different type of reporting, with different information and level of detail.”

According to Dijkstra, the often-cited benefit of more test coverage is misleading.

“That’s not depending on the framework, but on the people that use the framework to implement the automated checks [tests],” says Dijkstra. “They decide what is being tested. A good framework allows you to write [tests] effectively, and it might even help in showing code or requirements coverage, but it won’t [automatically] maximize coverage by itself.”

Qualified testers may develop the automation code (tests) in-house or use a combination of existing software automation tools to form the appropriate framework. The effort dedicated to developing automation code, or leveraging the automation tools that supplement the need for in-house code development, is maximized with the use of the appropriate test automation framework.  

Test automation frameworks are designed from the ground up by qualified professionals capable of programming automation code and customizing automation technology. This work requires experienced, technical testers and dedicated planning for the changes that may arise while implementing the automated tests. Testers also leverage and combine various automation tools to achieve a particular test objective.

Dijkstra points out that there is a buzz around “codeless test automation” solutions that commercial automation software companies are currently marketing. The claim is that there is no additional configuration or development required to implement automated testing.

“I don’t buy it,” says Dijkstra. “I’ve yet to see a test automation solution that both scales and is flexible enough, without the need for creating some sort of framework around it.” Either way, it’s imperative to align automation tools and frameworks with the testing goals. The goals for test automation decisions are rooted in the why.

Why Ask Why?

One of the most common mistakes associated with designing and implementing test automation is failing to ask the right questions and set realistic expectations.

“There are some good and some less good reasons for introducing test automation,” says Dijkstra. “Helping organizations [...] especially managers and C-level executives [...] become aware of what test automation can do for their software development process - shorten the feedback loop, for example - and what it highly likely will not do - deliver instant gratification, reduce testing headcount, save money from the get go - is a big first step toward realistic objectives.”

Dijkstra suggests asking the following questions before deciding on the type of test automation solutions to design or implement:

1. To what application tier can the automated tests be tied? This should be as close to the business or application logic being verified. For example, there is no need to use a user interface-driven test when the business logic is exposed through an API.
2. What would be the most effective test data management strategy? For example, are we relying on existing test data, or are we creating our test data at the start of every test run?
3. Do we have to deal with external and third party dependencies?
4. Are the dependencies readily available and do they contain the appropriate test data?
5. If not, can simulations (mocking, service virtualization) help us out here?
6. What should the reporting look like?
7. What information should be returned during and after running the automated tests?
8. What is the audience for the reports (this can be human or another system)?
9. What would be the most suitable programming language? The answer to this question should never be the defining choice, but it helps when there is some expertise around the chosen language in the organization. For example, there is no need to write a test automation framework in Python for a .NET shop, when there are viable C# alternatives.

After careful analysis of the specific results you desire and the objectives for the automated tests, you can decide what to automate. Then you move on to the how. Dijkstra writes:

“Only after all of the previous questions have been answered is it time to find the tool that best suits your requirements. Selecting a tool first and then trying to fit it into an existing software development and testing process is a bit like buying a horse and then realizing you live in an apartment on the twentieth story of a Manhattan skyscraper: probably not such a good idea.”

Once you arrive at the decision of how and are ready to leverage automation technology, you have plenty of choices. The rising popularity of test automation, combined with the fact that automated software tests are vital to Agile development creates a robust market for open source and commercial automation software tools.

There is an extensive list of existing software automation tools that testers combine based on their functionality to achieve their test automation goals. It is not always necessary to build a custom framework from the ground up. Depending on the development environment and the desired results, testers may leverage the best features of different tools as a framework solution. Here’s a sample of some popular options:

• Selenium: Selenium is a customizable, open source set of test automation tools for testing web applications in multiple languages. Selenium is popular and serves as the core technology that other open source browser automation tools, APIs, and frameworks “wrap around” to form hybrid automation frameworks.
• Robot Framework: Keyword-driven, Python-based framework for acceptance testing that runs on Java (Jython) and .NET (IronPython). It is open source and not dependent on specific operating systems or applications and can test more than websites.
• Serenity: An open source, Java-based framework, Serenity is designed to make writing automated acceptance and regression tests easier. Serenity is capable of producing illustrated reports with narratives to document an application’s activity.
• RedwoodHQ: An open source framework that works with all of the most common automation tools (Selenium, Appium, Silk, etc.), RedwoodHQ allows testers to develop test automation code using Java, Python, or C# and create keyword-driven test cases that are executed through a web interface.
• Sahi: Sahi is an open source framework for small requirements (less than 100 test scripts). It uses a record and playback function for test automation, works with all of the top web browsers (running Javascript), and it is compatible with Windows, Mac, and Linux. There is also a paid version of Sahi designed for web application testing
• Galen Framework: Galen is an open source test framework used for testing (JavaScript and Java tests) the layout and user experience design of web applications in a browser. It is designed for testing responsive websites and is based on the Selenium framework.
• SoapUI: SoapUI is a data-driven automation framework designed to test APIs. Open source and commercial versions are available.  
• HP UFT: Unified Functional Tester from HP is a test automation tool for functional and regression testing. Formerly known as QTP (Quick Test Professional), this software combines testing and keyword-driven framework capabilities. UFT works with web applications across major platforms and offers mobile test automation tools.
• C​ucumber: Automation code integration tool designed to support behavior-driven development method. Cucumber is an open source tool that merges test specification and documentation into a single source.

The linear automation framework is a first generation testing method for automated software testing and the simplest framework approach. Appropriately named, this framework guides software testing in a sequential fashion with little to no modification. This method is also known as “record and playback” and is used to test a web application’s user interface (UI). Testers record their step-by-step actions with the UI and then play back the recorded tests without focusing on reusing the test.

To highlight this approach, Bhargava provides an example of a sequence of actions for booking travel on a website in his book (2013) using Hewlett Packard’s popular software automation tool UFT (Unified Functional Testing), formerly known as QTP):

• Enter username
• Enter password
• Click on the OK button
• Enter the Flight date
• Select Fly From (source)
• Select Fly To (destination)
• Click on the FLIGHT button
• Click on the OK button
• Enter the name
• Click on the Insert Order button
• Close the Flight Reservation window

Even without the test code that accompanies Bhargava’s example using UFT, you can see the linear approach to testing the user interaction with a web application.

Advantages

• Quick: Using linear testing is the fastest way to generate a test with minimal planning involved.
• Intuitive: There is no programming or design expertise required to learn the record and playback feature of automation software.

Disadvantages

• Inefficient: The tests lack reusable functions, and the data is a part of the test so tests cannot run with multiple data sets.
• Problematic: Modifying the application makes maintenance complicated and error-prone (Bhargava, 2013).

Data-driven testing is a prominent practice in software testing. The use of data-driven automation frameworks evolved from the limitations of record and playback tools and methods that represented the first generation of automated testing technology. Authors Bruce Posey and Daniel Mosley describe data-driven frameworks as the third generation of automated test methods in their book, Just Enough Software Testing (2002).

Data-driven testing uses external data, such as an Excel spreadsheet file, as an input to control and verify the actions of an automated test. This data is developed independently of the test code. Data-driven test scripts are highly maintainable and easier for non-programmers to use. The benefit of this approach is the ability to modify test scenarios with code based on the data used. The data file contains the information required to test the application, so the test scripts do not need to be modified.

Advantages

• Efficient: Reduces the number of test code required to test scenarios and data is easier to modify.
• Reliable: Changes in the data do not impact the test code.
• Reusable: Improves test coverage with reusable tests.
• Flexible: Multiple data can execute tests only when required and by changing only the data inputs.

Disadvantages

• Investment: More upfront time and resource investment to set up the data source and functionality (that may end up changing).
• Expertise: Requires programming knowledge to develop test scripts that function appropriately.
• Complexity: Combination of external data sources can slow or crash software tools and make manual testing more complicated.

Keyword-driven testing is similar to data-driven testing. However, the test code delivers the data only. The data is not coded in the actual test; instead, it is located in an external data file. The code is the “key word” (where this framework gets its name). Posey and Mosley describe Keith Zambelich’s automated test framework in their book. Zambelich’s “test plan driven” method for automated testing used test cases containing “keywords” from a spreadsheet to control the processing. Zambelich points out that the entire process of his automated framework is data-driven (including the functionality), and the keywords control the processing.

Advantages

The same as Data-Driven Automation Frameworks, plus the following:

• Universal: Requires less technical knowledge and automation expertise after implementation. Keywords are reusable for multiple tests.
• Independence: A keyword-driven framework is not dependent on the application, automation tools, or programming languages and can perform manual testing activity as well.

Disadvantages

• Over Engineering: Requires an initial investment of time and resources to develop the framework (if only using a minimum amount of automated tests for a limited amount of time).
• Implementation: Initial learning curve for non-technical testers is high and requires more expertise to implement than to maintain time-to-market.

Data-driven and keyword-driven automation frameworks are a “generic scaffold” that guide automated testing of software applications. These frameworks evolved from the limitations of first-generation test automation technology. Modern software development cycles demanded faster software releases. According to Posey and Mosley, these frameworks provided the scaffolding for automating a larger number of functional tests using a minimal amount of test scripts.

The modular automation framework (“module-based” or “modularity framework”) divides automated test functions into logical, manageable segments, or modules. These modules are isolated to create independent tests that can then be recombined to follow a test plan. These modules of test code can be combined to form larger tests to represent functions of the different modules. According to Bhargava’s book, the crucial part of this framework design is decomposition and recombination.

Bhargava writes, “The key to achieving modularity is by decomposing the functionality and recombining the modules.”

The modular automation framework is also known as functional decomposition framework. It breaks down a user’s interaction with the application into smaller, independent tests, and then recombines these tests to achieve specific scenarios.  

Advantages

• Divide and Conquer: The complicated steps for designing and implementing automated tests are split into more manageable pieces to achieve results.
• Adjustable: Individual changes to one part of an application under test does not impact all tests for the application, only individual tests.
• Upkeep: Scalable division of maintenance for smaller modules of test data.

Disadvantages

• Complexity: The tests for each module contain embedded data so changes must be made to the tests if it requires different data sets.
• Aptitude: Requires more technical proficiency to integrate precise modules and design creativity is highly valuable to avoid substandard performance of the tests.  

Most test automation frameworks developed for modern software development cycles are hybrids. These frameworks combine the strengths of previous frameworks and try to avoid the weaknesses.

For example, a standard hybrid framework design would use the decomposition of test functions and a data-driven approach to organizing and managing resources with keywords (Bhargava, 2013). The advantage of using a hybrid framework is integrating the advantages of the other automation frameworks. The primary disadvantage is the increased complications that may arise from varying guidelines and the inability to scale across other applications under test.

In his experience, Dijkstra sees a hybrid approach with data and keyword-driven framework elements practiced most often.

“When I’m working on user interface-driven test automation (for example, using Selenium), I’ll take care of the data-driven part using a BDD tool such as Cucumber or SpecFlow and implement tests using the Page Object pattern, which can be considered a form of keyword-driven testing,” explains Dijkstra. “When I’m working on API-level test automation, my solutions are mostly data driven, since the scope and coverage of this type of tests [are] different.”

Other Test Automation Frameworks

• Business Process Testing (BPT) Framework: The BPT framework is a testing approach that divides business processes into small components that are reused for different tests. Similar to the modular automation framework, the goal is a collaborative approach to testing for various stakeholders.
• Test Library Framework: This is an “application-independent” framework. Similar to the modularity automation framework, the application under test is divided into a generic test library of functions instead of test scripts (Choosing a Test Automation Framework, 2003). These libraries contain the modules, sections, and functions of the application and are used to operate the test.

Choosing the appropriate test automation framework for your project comes down to deciding what guidelines will produce the desired results of the automated tests.

“What I see the most is people building a framework sort of from scratch, using open source tools (ex. Selenium) and applying accepted patterns such as the Page Object pattern, the Screenplay pattern, PageFactories, the LoadableComponent pattern, and so on,” says Dijkstra.

“That’s good because every team and every project has its own requirements [regarding] test behavior specification, test data management, reporting, error handling, etc.,” adds Dijkstra. “There’s no one best way, but there are patterns and abstraction techniques that make test automation code a lot more readable and maintainable and therefore make life a lot easier.”

When the automation framework and tools combine with common practices and coding standards for testing software, you have a set of standard guidelines in place to achieve your objectives for automating tests. It’s not always necessary to design and implement a custom framework, but Dijkstra cautions against the “tool-centric” approach that may lead to poor results when you fail to take the guidelines of an appropriate framework design into consideration.

He follows these steps to help his clients decide what approach to take when deciding on a test automation solution:

1. Discuss the Why: Dijkstra explains that determining why clients think they need test automation in the first place is important. He also gets a feeling whether it’s for the right or wrong reasons (according to him, reducing headcount or saving money up front is an example of the wrong reasons). From there, he works to set the proper expectations.
2. Ideal Approach: This is the what part of the decision making process. What needs to be automated? API-level automation? User interface-driven automation? What programming language is in use? Is the programming language a restriction?
3. Tools: Which tools would suit the organization’s needs best, then design an approach that fulfills the requirements. This is the how step.
4. Technical Proficiency: What is the level of technical expertise of the people that are going to extend and maintain the test automation?
5. Reporting: What type(s) of reporting does the automation need to deliver and to whom or where?
6. Data: What is the most effective way to deal with test data? Are we going to create it before every test? Is there a database backup we can restore before every test run

Test automation frameworks establish the guidelines to achieve the specific goals of automated tests. Automation frameworks are a solution that—when designed and implemented with the why, what, and how in mind—lead to the benefits of test automation. The automation framework should be easy to maintain and reusable. Open source software automation tools are available to use, and in some cases, can serve as an out-of-the-box solution to guide automated testing. You may prefer to have a commercial software test automation suite of tools provide the customer support and expertise. However, using a strict “tool-centric” approach to automation solutions is problematic. Rather, careful planning and consideration of the design and implementation of automation frameworks or the selection of software tools should involve a qualified test automation professional. A poorly designed, difficult-to-maintain automation solution limits the benefits of test automation.

Now that you understand all there is to know about test automation frameworks, it’s important to find a tool that enables you to track and manage the results of your tests. One such tool is Smartsheet, a collaborative work management platform that helps enterprises and teams work better.

 Use Smartsheet to track the schedule and results of planned, current, and completed tests. Share the schedule with your team and collaborate on the details in real time, in one central location. Whether you’re running manual or automated tests, Smartsheet’s broad range of views – Calendar, Gantt, Card, and traditional Grid – allow you to manage progress the way you want. Organize test results with hierarchy and use comments to keep work in context.

 Plus, with Smartsheet Sights you can create custom dashboards to monitor testing and provide a high-level view of progress and key metrics to management. Improve collaboration on the testing process, increase control of timing and resources, and gain unprecedented visibility into results with one location for the truth.

See how easy it is to track and manage automation testing using a template in Smartsheet.

 Track Automation Testing in Smartsheet