Automated Software Testing Isn't Automatic: Introduction to the Basics

Before attempting to define software automation testing in concrete terms, it’s important to provide some context. If you ask Google, the search results offer a variety of descriptions. Here are a few key themes:

• There are interchangeable ways to refer to software automation testing including test automation, automation testing, automated software testing, automated testing, testing automation, automated testing for software, and more.
• Test automation involves using software tools to test software.
• There is a difference between manual testing and automation testing - they are both necessary for testing software today.
• How much manual intervention should be involved when automating software tests is open for debate.
• Automation testing leverages the computing power of machines to outperform the physical and mental task output of people.
• Test automation is essential to modern software development practices because of the frequency of new releases.
• The people that use test automation technology have different levels of technical proficiency, experience, and training.

Now that you have a snapshot of various themes and terms used to describe automation testing, there is an important distinction to be made:

Testing Software: An investigation performed by a human to provide project stakeholders with information about the quality of the software under development.

Automation Testing: The use of software tools (or automation code) and the power of machines to control the execution of tests, compare their outcomes, and report functions that would otherwise require manual testing activity. This is sometimes called test automation.

It is easier to understand the difference if you first recognize that test automation isn't automatic. Or, as Jim Hazen, an Automation Consultant and veteran of the software testing trenches is fond of saying, “It’s automation, not automagic.”

Hazen uses the term “automagic” to get people to think about what their goals are for using automation tools and technology for their specific project needs. He cautions against assuming the use of automation testing tools is a cure-all or silver bullet solution. As Hazen points out, automation testing is still dependent on the people performing the testing.

“Basically, (test) automation is only as smart as the person who built it,” says Hazen.

There is a common reference to a “shift left” approach in modern development practices. This term refers to the advent of testing software earlier in the development cycle than traditional methods. Developers are now responsible for, and held accountable to, testing their code as they create it (sometimes before it's developed, but more on that later). Also, test professionals capable of a higher level of technical expertise, including the ability to write code (automation code), are in demand and job titles often go by a variety of names.

The jobs section on LinkedIn lists the following search results for “automation” in the United States:

This article uses the term “tester” to refer to the person involved in testing software with automation tools. It is not meant to distinguish by job title or technical proficiency. Jim Hazen describes himself as a hybrid, or “technical tester,” because he can write test scripts and develop what he refers to as “testware.” The trend is to hire for multiple skillsets, but that does not mean the non-technical stakeholders involved in software development don’t benefit from automation testing.

“Tester or Test Engineer or SDET or Automation Developer...a rose by any other name in my book,” says Hazen.

He prefers to use the term “automated test execution” when discussing test automation because the majority of people are referring to automating that activity in the testing process. Non-technical testers should have access to the automation tools. Today’s modern automation technology makes it possible for teams to collaborate and benefit from automated testing.  

According to Hazen, there are various ways to use automation tools to automate the testing process. His list includes:

• Data modeling/generation
• Scenario modeling
• Analysis (scanning log files for example)
• Reporting
• Utilities (setup/clean up test environment)
• Test management
• Test execution statistics (pass/fail, functionality covered)

“It is all about tooling, and how you apply tools to aid in the work itself,” explains Hazen.

Automated testing is different than “automatic manual testing,” according to author and consultant James Bach. It is not a computer performing the same task without the help of a human brain. Instead, Bach defines test automation as the use of any tools to aid testing.

It’s best to avoid comparing automated and manual testing as if they are exclusive of one another. Both are used to test software. Testers design the various automated tests that are used to automate workflows based on what they want to check. Depending on their programming ability, testers may even write their own “in-house” automation code.

The takeaway is that testing is a process requiring human intervention. Bas Dijkstra, an experienced test automation consultant, describes how even the term “test automation” is flawed unless you understand what is and isn’t automated. The actual “learning, exploring, and experimenting” involved in manual, human-performed testing cannot be automated, according to Dijkstra. He writes:

“I don't think that using the 'test automation' label in itself is wrong though, as long as people are aware of what is being automated (checks) and what is not (tests). This difference between testing and checking also provides an argument as to why manual testing as an activity will not cease to exist, at least not for the foreseeable future: testing activities cannot be automated!”

Alan Page is an author with more than two decades of experience in software testing roles, the majority spent in various roles at Microsoft. He offers another perspective on the importance of distinguishing automated and manual testing. In “The A Word,” an ebook compilation of his blog posts on automation, Page mentions that most of his commentary on automation focuses on the “abuse and misuse” of automation in software testing and development. He is skeptical of replacing manual testing activity with test automation, as you can see from the his Twitter feed:

There are many goals often cited by the test community and vendors selling test automation products and services. These goals include:

• Simplifying test execution
• Increasing speed of executing tests (test scripts)
• Increasing the amount of test coverage
• Improving the reliability of tests
• Shortening software development cycles
• Minimizing human interaction with testing
• Reducing maintenance cost of testing
• Improving accuracy of software tests
• Saving time and money
• Improving team morale
• Developing software value
• Eliminating boring tasks

Hazen believes marketing efforts and various misinformation surrounding test automation tools (software products) often give decision makers misguided expectations about the benefits of test automation. He sees automation solutions over and under-engineered depending on the specific goals of a project.

“The purpose (and primary benefit) of automation testing is efficiency gains,” says Hazen. “A byproduct of this efficiency is allowing your testers to focus on the higher value tasks.”

“You leverage the power of machines and the knowledge of qualified testers to provide economies of scale that would otherwise be impossible for humans to achieve manually,” adds Hazen.  

Test Automation at Google

The increased level of production is important to companies developing software for rapid (sometimes daily) release. Companies like Google automate testing to scale their software development process and release products that billions of users rely on daily. Google created new testing roles and job titles for their engineers when they realized the benefits of automated testing during their rapid growth. Their efforts resulted in higher quality, more reliable, and more frequently released software.

The use of automation testing is a response to the demand for continuous innovation and the need for rapid software releases. That does not mean it is a modern technology, only that the tools and methods have evolved. Automated software testing was around from the beginning.

The origins of test automation start with the computing industry. The book, Automated Software Testing: introduction, management, and performance, notes that the history of automated software tests followed the evolution of software development. Software testing in the era of large database systems that supported scientific and government programs meant that a finite amount of test procedures could test a complete system at the end of the development cycle. With the rise of personal computing, the methods for testing software changed to keep up with increased demand for new software applications and new product features.

A report cited in the book found that software developers in the 1990s routinely missed ship dates and deadlines. The pressure to reduce costs and keep up with the demands of a rapidly changing market is now dependent on faster software development. With growth and competition in commercial software development came new technology that changed software forever. The new graphical user interface (GUI), networked personal computers, and the client-server architecture demanded new development and testing tools.

The use of GUI applications introduced the first generation of automated test tools capable of performing record and playback functions. Testers continued to write down scenarios and test scripts, but the widespread use of GUI meant that users of an application now had multiple ways to interact with the software. Testers had to overcome this scenario, and the evolution of test automation tools gained momentum.

To do more with less, developers reused test scripts during development and integration stages to work more efficiently. The demand for new software built, and the constant change to software under development opened the door for automation testing practices to serve as a reliable control mechanism for testing the code (Automated Software Testing, 1999).

The expansion of automated testing practices continued with the rise of modern development methods such as Agile software development. While test automation is successful in traditional development, it is vital to Agile and the evolution of various practices associated with the methodology.

The titans of information technology — Google, Amazon, and Facebook — develop software at a rapid pace to meet the demand for their products and services. In many cases, the teams responsible for product development at these companies adopt Agile practices to achieve their goals.

Test automation is a fundamental part of Agile. Various core practices of Agile, such as Continuous Integration (CI), Continuous Delivery, Test-Driven Development (TDD), and Behavior-Driven Development (BDD) rely on the efficiency and reliability of test automation. For teams using Agile methods, test automation impacts more than just the software being developed: successful test automation practices also highlight the culture change and importance of teamwork associated with Agile.

Angie Jones is an Automation Engineer, Professor, Inventor, and Consultant with experience leading multiple “in-sprint” Agile teams. Jones says that within the last five years, test automation is more mainstream because of the growth and adoption of Agile practices in software development.

“With Agile methodology, as a culture, we need to move faster,” explains Jones. “We don’t have time for redundant testing of the whole regression bucket manually. We cannot afford to do testing the way we did before.”

Jones believes the most common reason for using test automation today is to shorten the regression test cycle. Regression tests are used to determine if changes to the software are the cause of new problems. They verify that a system under test hasn’t changed. To guard against introducing unintended changes, they become part of a regression test suite after the tests pass. Regression tests are automated to ensure regular feedback.

“In more advanced projects, the goal is to incorporate test automation into the continuous integration process,” adds Jones.

Martin Fowler, author, consultant, and thought leader on various software development topics, describes Continuous Integration (CI) as a practice that leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly (often daily). Automated tests are fundamental to this practice. Fowler writes:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, (sic) usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.”

Once the software passes automated tests, it may be released into production (depending on the preferred rate of deployment). This process is called Continuous Delivery. The preferred frequency is the difference between Continuous Delivery and Continuous Deployment. You achieve Continuous Delivery with the steps required for CI. The emphasis on automated testing (and automated builds) for quality assurance capitalizes on the efficiency of successful test automation and is essential to this practice.

You need collaboration and extensive automation to achieve Continuous Delivery. According to Fowler, the rewards of doing so successfully include reduced risk, believable progress, and user feedback. Continuous Delivery is an important method in Agile development. It helps remove obstacles that prevent the frequent deployment of features. Automation testing is a fundamental part of the continuous development practice associated with Agile.  

Risk and Reward

As it relates to testing software, Hazen looks at Agile and non-Agile methods of development as being risk-based decisions. According to Hazen, the question of how test automation impacts Agile or other development methods comes down to how much automation “tooling” is used, where it is implemented in testing, and how much it is relied on for the project’s goal.  

Automated testing expanded with Agile principles because testing in a repeatable manner that is secure, reliable, and keeps pace with the rapid deployment of software is required for this environment. In their book Agile Testing: A Practical Guide for Testers and Agile Teams, authors Lisa Crispin and Janet Gregory claim Agile development depends on test automation to succeed. They emphasize the team effort required for test automation and recommend automating tests early in the development process. Also, the development of automation code is as important as the development of the actual production code for software. The “test-first" approach to development is known as Test-Driven Development.

Crispin and Gregory define Test-Driven Development (TDD) as the process of writing and automating small unit tests before writing the piece of code that will make the test pass. TDD is used for continuous integration testing to ensure small units of code work together first. A unit test verifies the behavior of a small part of the code in the overall system. These tests are the primary candidate for the majority of automated tests. Even teams that are not practicing Agile development use TDD to prevent defects and design software (Agile Testing, 2008).

TDD is misleading if you don’t realize that it is more about software design and teamwork than testing. According to the authors, an Agile programmer using TDD to write “test-first” code can think about what functionality they want from the code and then partner with a tester to make sure all aspects of the code are performing to that standard of functionality.

Chandra Kandukuri is a Technical Test Lead at Microsoft with more than 16 years of software development experience in multiple environments, developing automation frameworks and tools. He advocates the use of TDD and dedicating the time and resources to do it well. Although it is relatively uncommon to see teams utilize TDD in his experience, Kandukuri recommends the method with automated software testing because of the positive teamwork habits it can promote.

“Automation encourages people to save time and [makes] work more reliable and efficient. Testers become natural problem solvers, and [some] steps can be removed that are not important or [not] required for other members of the team,” says Kandukuri.

Dan North, a technology consultant with a background in Agile software development, created BDD in response to the confusion he experienced while teaching test-driven development. North writes:

“While using and teaching Agile practices like test-driven development (TDD) on projects in different environments, I kept coming across the same confusion and misunderstandings. Programmers wanted to know where to start, what to test and what not to test, how much to test in one go, what to call their tests, and how to understand why a test fails. [….] My response is BDD.”

Jones defines BDD as the process where teams use domain-specific language to express the expected behavior of an application through scenarios. She points out that this is not magic - there is automation code involved in the process - but that BDD is ideal for developers and testers sharing automation work. Specialized tools like Cucumber, the most popular open source tool for automation code integration, executes this work and is the tool of choice for Jones.

The IT industry depends on similar Agile practices of different names to meet the market’s demand for their products and services. Test automation is vital to Agile and the companies using Continuous Integration and Delivery, TDD, and BDD. For the titans of technology and the IT industry at large to reap the benefits of test automation, they must rely on automation frameworks.

A test automation framework is a set of guidelines used to produce beneficial results of the automated testing activity. These guidelines may include:

• Common practices
• Assumptions for the desired outcome
• Test tools (software) and interfaces
• Test libraries
• Coding standards

Automation frameworks, when designed and implemented correctly, deliver frequent and stable automated test code. With a proper framework, the code is easier to maintain and often reusable.

“When we refer to automation frameworks, it is easiest to understand with the functional testing areas,” says Kandukuri. “You are providing commonly used methods to improve the efficiency of automated tasks. With limited knowledge of how the test case is set up, a tester can fall back on the framework to refer to simple statements and implement the test cases.”

According to Kandukuri, automation framework means different things to people depending on the context. For this reason, he does not start with the big picture when developing frameworks.

“For regression testing, you may begin automating tests for only the most critical, basic functions,” says Kandukuri. “You should not isolate frameworks. It should be easy to maintain and use.”

Automation Framework: Strategies and Tools

Jones defines automation frameworks in concrete terms. “It is the foundation for a successful automation code base. Your test code will evolve with the framework. If the automation framework is wrong for the project goals, it is not maintainable, harder to manage, and you risk design errors,” she says.

Automation frameworks are combined with specific automation tools to create a sound basis for your specific project goals. Automation tools are then aligned with testing goals. When the framework and tools are combined with common practices and coding standards for testing software, you have an automation framework. Jones offers an example using the most popular open source automation technology used for testing a web browser’s user interface (UI).

“Selenium is a technology used for automating browser actions. There is no verification or opinion, true or false, pass or fail. You need to combine Selenium with a verification tool, and possibly another tool for reporting results,” explains Jones.

Jones recommends flexible automation frameworks and cautions against using a framework limited to only UI testing, for example. Some test teams build their frameworks from scratch to satisfy the desired result of the test automation code and activities. According to Jones, most test automation initiatives fail due to the poor design of the test automation framework architecture for that project.

“If you need a framework to test web services, you may use a different set of tools within a framework,” says Jones. “You should be able to combine tools within a framework in a way that allows you to test, so you are not limited to just UI, integration, or web-services testing. Build your framework in a way that supports a range of testing goals.”

Hazen describes automation frameworks as ways of building in maintainability and reusability of the automated tests, which are some form of automation code. He believes this is the key to successful automation.  

“Too many projects/implementations are messes because they don't plan or design the implementation,” says Hazen. “Building automation is the same as building application code. Automation is a form of software development.”

Choosing the framework for your project comes down to deciding what guidelines will produce the desired results of the automated tests. Often, developers end up designing a custom framework. This requires experienced testers and dedication to planning for the changes that may arise while implementing the automated testing. In some cases, an existing automation tool already has the functionality necessary to achieve the desired result of automated tests.  

Types of Automation Frameworks:

• Linear Scripting Framework: Recording and replaying test scripts in sequential (“linear”) fashion with little or no modification.
• Data-driven Framework: A constant source of test criteria (internal or external data) specifies the test scripts to run.
• Keyword-driven Framework: Tables on a spreadsheet specify the action of a test script based on a library of functions for an assigned keyword.
• Modular Testing Framework: Modules of an application under test are divided and tested with individual test scripts that can be combined for larger test scripts.  
• Hybrid Testing Framework: A combination of frameworks to leverage the strengths of each.

Automation frameworks provide guidelines to achieve beneficial results from test automation tools and automated testing activity. They establish a universal standard for testers to achieve the specific goals of the automated tests. The framework should be easy to maintain and easy to change. Consider dedicating the role of framework design and development to a dedicated, qualified tester. A poorly designed — or hard to maintain — framework causes problems even if you are using the right automation software tools. Poor planning and the failure to create or select the appropriate framework to guide test automation activity limits the benefits of automating tests.

Further details on how to compare these frameworks, including guidelines for design and how to select the best option. 

Angie Jones

Angie Jones is a Consulting Automation Engineer who advises several Scrum teams on automation strategies and has developed automation frameworks for many software products. Angie speaks and teaches internationally at software conferences, serving as an Adjunct College Professor of Computer Programming, and also teaches tech workshops to young girls through TechGirlz and Black Girls Code. Find out more on LinkedIn and at angiejones.tech

On Test Automation Strategy:

“Just because you can code it doesn't mean it should be automated.”

“The most important thing to consider is the problem you are trying to solve. Many test automation initiatives fail because teams are trying to jump in head first and automate every test possible instead of the most valuable tests according to the goals of development. They find themselves in a maintenance nightmare. Pick the most valuable test you were already performing manually and automate those first.”

On Automation Tools and Technology:

“Automation code is as important as production code for an application.”

“Selenium is the go-to UI automation tool. The other credible open source tools are essentially a wrap-around tool around Selenium. For web service testing, I prefer REST Assured. SoapUI is another option used frequently and offers a professional version in addition to open source. Testing G and Junit are popular for verification tools. For BDD, Cucumber and Specflow are popular with the Microsoft stack of development tools.”

On Continuous Integration:

“When a team is running continuous integration you need fast feedback from testing. If you automate everything and have thousands and thousands of automation tests that take hours to run, you risk ruining the original goal of automation testing.”

On Test Automation Challenges:

“A common pitfall is deciding what resource will be responsible for automation testing. Your development team that is responsible for production code may be unqualified or short on time to successfully automate testing. It goes both ways: your [current] testers may be unqualified to write good automation code to realize the benefits.”

“Another common mistake is trying to get testers to do both jobs, so when management gives the go ahead for automation testing, any QA related job these days requires some level of automation and testers might get excited about the potential for test automation. But these are both full-time jobs, so often times [these] teams struggle with deciding what to spend limited time on.”

Jim Hazen

Jim Hazen is an Automation Consultant and “veteran of the software testing trenches” who helps companies with test automation and performance test implementations. He has presented at multiple professional conferences, including STARWest and STPCon, and published articles in ST&QA Magazine on test automation and communication techniques for testers. You can learn more about Jim on LinkedIn.
 

On Test Automation Strategy:

“Automation is not the end-all solution; it is a piece of the puzzle. It will not solve your problems; it has the potential to create new ones. How well you use it is up to your planning and process for using test automation.”

“First thing I do is figure out who the players are and what they understand about the work to be done. I do that so I can fix the myths and misconceptions (of test automation) early and quickly. I work to get people on the same page and with the same understanding.”

On What to Automate:

Hazen views this question as a risk-based approach. He categorizes a client’s current tests based on what they are testing and prioritizing when it needs to run - his technique will determine what automated tests to build first. He uses three basic “buckets” to categorize tests and avoid a shotgun approach to testing:

1. Smoke Tests: Smoke tests, or build verification tests, are a series of tests run to verify an application is running correctly before moving on to more tests. If these tests don’t pass you do not move forward with time and resources.
2. 80/20 Rule: Jim uses Pareto’s Law to ask “what 20 percent of the application is used 80 percent of the time.” A variety of deeper testing here focuses on what tests are mandatory to achieve desired results on “high risk” items.
3. Other: Ask What remaining peripheral tests are left that prevent the application from release? or What is broken? These tests are for the medium-low priority items.

This exercise helps to keep the focus on what needs to be done to see benefits as soon as possible. Hazen notes that this process leads to efficiency gains, his primary goal for automated test execution.

On Test Automation Challenges:

“Companies frequently over-estimate what their existing development teams can accomplish with automation tools. They often end up bringing in an automation consultant after realizing their existing team is under-qualified to realize the full benefits of test automation.”

Chandra Kandukuri

Chandra Kandukuri is a Technical Test Lead at Microsoft with more than 16 years of software development experience in multiple environments, including over 13 years in various test roles including Sr. SDET. He develops automation frameworks and has experience using a host of automated tools and technology. Learn more about Chandra on LinkedIn.

On Hiring for Automation Testing:

“Hire for automation framework development. If a tester is not qualified to automate tasks he will ask the framework developer to come up with automated tests for new features as they are developed. This [development] would be the dedicated resource’s primary job.”

As projects evolve, you may run out of time or the resources required to keep up with the demands for automation testing. Kandukuri recommends hiring professionals who can develop automation frameworks and separate testers responsible for automated test execution.

“Have the person developing the automation framework [be different] from the testers executing the tests. You are paying a resource actually to save time and money.”

On What to Automate:

“We always plan to automate mandatory test cases. The non-mandatory test cases are not automated. You know that for thousands of regression tests, hundreds will fail.”

When he has time away from the mandatory test cases, Kandukuri can address these failures, but the development cycle does not slow down. He uses automation testing to address the issues.  

“It is key to automate regression testing. Automate regression tests for high priority features, then make it mandatory for a minimum validation of unit tests and determine this metric.”

Kandukuri estimates that about 50 percent of his time is dedicated to automation tasks. If there is a feature release, he uses manual testing, and then works to automate those tests. The goal is to focus on the new feature changes while the previous feature tests are automated.  

“There are millions of regression tests for a Windows 10 release. For example, if you plan 10 new features, five [of those 10] are critical and a priority. These test cases will be the criteria used to release the software. You build from that progress. So on the next release, you have new features, 10 are determined critical for testing. So it keeps adding, now you have 15 regression tests being automated to keep up with the release schedules.”

Realizing the benefits of software automation testing first requires understanding that automation isn’t automatic. If you understand the basics — what it is, what it is not, who uses it and why they do so — you will start to see why automation testing is fundamental to modern software development. The efficiency gains associated with successful test automation require the use of automation frameworks and proper automation software tools.

Here are some more resources to explore to learn more about automated testing:

Blogs and Websites:

• Angie Jones Blog
• Google Testing Blog
• Ministry of Testing
• Martin Fowler Blog
• OnTestAutomation.com
• TechBeacon
• Automation Awesomeness
• Tooth of the Weasel
• James Bach Blog

Podcasts:

• Test Talks with Joe Colantonio
• STP Radio
• The Testing Show
• Let’s Talk About Test, Baby

Conferences:

• Software Testing Conference List
• Software Test Professionals (STP) Conference
• Ministry of Testing: TestBash Conference
• QUEST Quality Engineered Software and Testing Conference
• STARWEST Software Testing Conference

Now that you understand the basics of software automation testing, it’s important to find a tool that enables you to track and manage the results of your tests. One such tool is Smartsheet, a collaborative work management platform that helps enterprises and teams work better.

 Use Smartsheet to track the schedule and results of planned, current, and completed tests. Share the schedule with your team and collaborate on the details in real time, in one central location. Whether you’re running manual or automated tests, Smartsheet’s broad range of views – Calendar, Gantt, Card, and traditional Grid – allow you to manage progress the way you want. Organize test results with hierarchy and use comments to keep work in context.

 Plus, with Smartsheet Sights you can create custom dashboards to monitor testing and provide a high-level view of progress and key metrics to management. Improve collaboration on the testing process, increase control of timing and resources, and gain unprecedented visibility into results with one location for the truth.

See how easy it is to track automation testing using a template in Smartsheet.

Track Automation Testing in Smartsheet