Compare with Manual testing

Manual testing

Manual testing is the process of manually testing software for defects. It requires a tester to play the role of an end user, and use most of all features of the application to ensure correct behavior...

.

Test automation is the use of software to control the execution of test

Software testing

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...

s, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.

Overview

Although manual tests may find many defects in a software application, it is a laborious and time consuming process. In addition, it may not be effective in finding certain classes of defects. Test automation is the process of writing a computer program to do testing that would otherwise need to be done manually. Once tests have been automated, they can be run quickly and repeatedly. This is often the most cost effective method for software products that have a long maintenance life, because even minor patches over the lifetime of the application can cause features to break which were working at an earlier point in time.

There are two general approaches to test automation:

• Code-driven testing. The public (usually) interfaces
  Public interface
  A public interface is the logical point at which independent software entities interact. The entities may interact with each other within a single computer, across a network, or across a variety of other topologies...
  
   to classes, modules or libraries are tested with a variety of input arguments to validate that the results that are returned are correct.
• Graphical user interface
  Graphical user interface
  In computing, a graphical user interface is a type of user interface that allows users to interact with electronic devices with images rather than text commands. GUIs can be used in computers, hand-held devices such as MP3 players, portable media players or gaming devices, household appliances and...
  
   testing. A testing framework generates user interface events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.

Test automation tools can be expensive, and it is usually employed in combination with manual testing. It can be made cost-effective in the longer term, especially when used repeatedly in regression testing

Regression testing

Regression testing is any type of software testing that seeks to uncover new errors, or regressions, in existing functionality after changes have been made to a system, such as functional enhancements, patches or configuration changes....

.

One way to generate test cases automatically is model-based testing

Model-based testing

Model-based testing is the application of Model based design for designing and optionally executing the necessary artifacts to perform software testing. Models can be used to represent the desired behavior of the System Under Test , or to represent the desired testing strategies and testing...

 through use of a model of the system for test case generation but research continues into a variety of alternative methodologies for doing so.

What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make. Selecting the correct features of the product for automation largely determines the success of the automation. Automating unstable features or features that are undergoing changes should be avoided.

Code-driven testing

A growing trend in software development is the use of testing frameworks such as the xUnit

XUnit

Various code-driven testing frameworks have come to be known collectively as xUnit. These frameworks allow testing of different elements of software, such as functions and classes...

 frameworks (for example, JUnit

JUnit

JUnit is a unit testing framework for the Java programming language. JUnit has been important in the development of test-driven development, and is one of a family of unit testing frameworks collectively known as xUnit that originated with SUnit....

 and NUnit

NUnit

NUnit is an open source unit testing framework for Microsoft .NET. It serves the same purpose as JUnit does in the Java world, and is one of many in the xUnit family.- Example :Example of an NUnit test fixture:using NUnit.Framework; [TestFixture]...

) that allow the execution of unit test

Unit test

In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use.A unit is the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function...

s to determine whether various sections of the code

Code

A code is a rule for converting a piece of information into another form or representation , not necessarily of the same type....

 are acting as expected under various circumstances. Test case

Test case

A test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...

s describe tests that need to be run on the program to verify that the program runs as expected.

Code driven test automation is a key feature of Agile software development

Agile software development

Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams...

, where it is known as Test-driven development

Test-driven development

Test-driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new...

 (TDD). Unit tests are written to define the functionality before the code is written. Only when all tests pass is the code considered complete. Proponents argue that it produces software that is both more reliable and less costly than code that is tested by manual exploration. It is considered more reliable because the code coverage is better, and because it is run constantly during development rather than once at the end of a waterfall

Waterfall model

The waterfall model is a sequential design process, often used in software development processes, in which progress is seen as flowing steadily downwards through the phases of Conception, Initiation, Analysis, Design, Construction, Testing, Production/Implementation and Maintenance.The waterfall...

 development cycle. The developer discovers defects immediately upon making a change, when it is least expensive to fix. Finally, code refactoring is safer; transforming the code into a simpler form with less code duplication, but equivalent behavior, is much less likely to introduce new defects.

Graphical User Interface (GUI) testing

Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. The advantage of this approach is that it requires little or no software development. This approach can be applied to any application that has a graphical user interface. However, reliance on these features poses major reliability and maintainability problems. Relabelling a button or moving it to another part of the window may require the test to be re-recorded. Record and playback also often adds irrelevant activities or incorrectly records some activities.

A variation on this type of tool is for testing of web sites. Here, the "interface" is the web page. This type of tool also requires little or no software development. However, such a framework utilizes entirely different techniques because it is reading HTML instead of observing window events.

Another variation is scriptless test automation that does not use record and playback, but instead builds a model of the Application Under Test (AUT) and then enables the tester to create test cases by simply editing in test parameters and conditions. This requires no scripting skills, but has all the power and flexibility of a scripted approach. Test-case maintenance seems to be easy, as there is no code to maintain and as the AUT changes the software objects can simply be re-learned or added. It can be applied to any GUI-based software application. The problem is the model of the AUT is actually implemented using test scripts, which have to be constantly maintained whenever there's change to the AUT.

What to test

Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.

One must keep satisfying popular requirements when thinking of test automation:

• Platform and OS
  Operating system
  An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
  
   independence
• Data driven capability (Input Data, Output Data, Metadata
  Metadata
  The term metadata is an ambiguous term which is used for two fundamentally different concepts . Although the expression "data about data" is often used, it does not apply to both in the same way. Structural metadata, the design and specification of data structures, cannot be about data, because at...
  
  )
• Customizable Reporting (DB Access, crystal reports
  Crystal Reports
  -External links:*****Reviews:****...
  
  )
• Easy debugging and logging
• Version control friendly – minimal binary files
• Extensible & Customizable (Open APIs to be able to integrate with other tools)
• Common Driver (For example, in the Java development ecosystem, that means Ant
  Apache Ant
  Apache Ant is a software tool for automating software build processes. It is similar to Make but is implemented using the Java language, requires the Java platform, and is best suited to building Java projects....
  
   or Maven
  Apache Maven
  Maven is a build automation and software comprehension tool. While primarily used for Java programming, it can also be used to build and manage projects written in C#, Ruby, Scala, and other languages. Maven serves a similar purpose to the Apache Ant tool, but it is based on different concepts and...
  
   and the popular IDEs
  Integrated development environment
  An integrated development environment is a software application that provides comprehensive facilities to computer programmers for software development...
  
  ). This enables tests to integrate with the developers' workflows.
• Support unattended test runs for integration with build processes and batch runs. Continuous integration
  Continuous integration
  In software engineering, continuous integration implements continuous processes of applying quality control — small pieces of effort, applied frequently...
  
   servers require this.
• Email Notifications (automated notification on failure or threshold levels). This may be the test runner or tooling that executes it.
• Support distributed execution environment (distributed test bed
  Testbed
  A testbed is a platform for experimentation of large development projects. Testbeds allow for rigorous, transparent, and replicable testing of scientific theories, computational tools, and new technologies.The term is used across many disciplines to describe a development environment that is...
  
  )
• Distributed application support (distributed SUT
  System under test
  System under test refers to a system that is being tested for correct operation. The term is used mostly in software testing.A special case of a software system is an application which, when tested, is called an application under test....
  
  )

Framework approach in automation

A framework is an integrated system that sets the rules of Automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort.

Defining boundaries between automation framework and a testing tool

Tools are specifically designed to target some particular test environment. Such as: Windows automation tool, web automation tool etc. It serves as driving agent for an automation process. However, automation framework is not a tool to perform some specific task, but is an infrastructure that provides the solution where different tools can plug itself and do their job in a unified manner. Hence providing a common platform to the automation engineer doing their job.

There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are:

1. Data-driven testing
   Data-driven testing
   Data-driven testing is a term used in the testing of computer software to describe testing done using a table of conditions directly as test inputs and verifiable outputs as well as the process where test environment settings and control are not hard-coded...
   
   
2. Modularity-driven testing
   Modularity-driven testing
   Modularity-driven testing is a term used in the testing of software.-Test Script Modularity Framework:The test script modularity framework requires the creation of small, independent scripts that represent modules, sections, and functions of the application-under-test...
   
   
3. Keyword-driven testing
   Keyword-driven testing
   Keyword-driven testing, also known as table-driven testing or action-word testing, is a software testing methodology for automated testing that separates the test creation process into two distinct stages: a Planning Stage, and an Implementation Stage.-Overview:Although keyword testing can be used...
   
   
4. Hybrid testing
   Hybrid testing
   -Overview:The hybrid Test Automation Framework is what most frameworks evolve into over time and multiple projects. The most successful automation frameworks generally accommodate both Keyword-driven testing as well as Data-driven testing....
   
   
5. Model-based testing
   Model-based testing
   Model-based testing is the application of Model based design for designing and optionally executing the necessary artifacts to perform software testing. Models can be used to represent the desired behavior of the System Under Test , or to represent the desired testing strategies and testing...
   
   

Notable test automation tools

Tool name	Produced by	Latest version
HP QuickTest Professional HP QuickTest Professional HP QuickTest Professional software provides functional and regression test automation for software applications and environments. Part of the HP Quality Center tool suite, HP QuickTest Professional can be used for enterprise quality assurance....	HP Software Division	11.0
HTTP Test Tool HTTP Test Tool HTTP Test Tool is an open source scriptable protocol test tool for HTTP protocol based products like web servers, web browsers, web applications or ICAP....	Open source Open source The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...	2.0.8
IBM Rational Functional Tester IBM Rational Functional Tester IBM Rational Functional Tester is a tool for automated testing of software applications from the Rational Software division of IBM. It allows users to create tests that mimic the actions and assessments of a human tester...	IBM Rational Rational Software Rational Machines was founded by Paul Levy and Mike Devlin in 1981 to provide tools to expand the use of modern software engineering practices, particularly explicit modular architecture and iterative development...	8.2.0.2
LabVIEW LabVIEW LabVIEW is a system design platform and development environment for a visual programming language from National Instruments. LabVIEW provides engineers and scientists with the tools needed to create and deploy measurement and control systems.The graphical language is named "G"...	National Instruments	2011
Maveryx Maveryx Maveryx is an automated functional, graphical user interface , and regression test tool for Java applications. Differently from other test tools Maveryx does not use any ‘GUI Map’ to create and run the automated tests. GUI objects to test are recognized at execution time, by an advanced GUI Objects...	Maveryx	1.2.0
QF-Test QF-Test QF-Test from Quality First Software is a cross-platform software tool for the GUI test automation limited to Java/Swing, SWT, Eclipse plug-ins and RCP applications, ULC, Java applets, Java Web Start and cross-browser test automation of web-based applications .- Overview...	Quality First Software GmbH	3.4.3
Ranorex Ranorex Ranorex is a Windows GUI test automation framework for testing many different application types including Web 2.0 applications, Win32, MFC, WPF, Flash/Flex, .NET and Java . Ranorex doesn't have a scripting language of its own. The user Ranorex is a Windows GUI test automation framework for testing...	Ranorex GmbH	3.1.2
Rational robot	IBM Rational	2003
Selenium Selenium (software) Selenium is a portable software testing framework for web applications. Selenium provides a record/playback tool for authoring tests without learning a test scripting language . It also provides a test domain-specific language to write tests in a number of popular programming languages, including...	Open source Open source The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...	2.11
SilkTest SilkTest SilkTest is an automation tool for testing the functionality of enterprise applications in most versions of Windows, Solaris 9 and 10, and Red Hat Enterprise Linux WS 2.1 and 3.0. It was produced by Segue Software which was acquired by Borland in 2006. Borland was acquired by Micro Focus...	Micro Focus Micro Focus International Micro Focus International plc is a multinational software and information technology business based in Newbury, Berkshire, United Kingdom. The firm provides software and consultancy services for clients updating legacy systems to more modern platforms...	2010 R2 WS2
SOAtest Soatest Parasoft SOAtest is "a comprehensive testing and analysis tool suite for tailored to the unique testing and validation needs of Service Oriented Architectures." It helps testers test the message layer functionality of their services by automating their testing and supports numerous transport...	Parasoft Parasoft Parasoft is an independent software vendor with headquarters in Monrovia, California. It was founded in 1987 by five graduates of the California Institute of Technology who had been working on Caltech Cosmic Cube....	9.0
TestComplete TestComplete TestComplete is an automated testing tool, developed by SmartBear Software which aims to allow testers to create software quality tests. Tests can be recorded, manually scripted or created manually with keyword operations and used for automated playback and error logging.TestComplete is used for...	SmartBear Software SmartBear Software SmartBear Software is a software company which provides tools for software developers, quality assurance engineers and IT professionals. SmartBear Software is the unification of AutomatedQA, the original Smart Bear Software and Pragmatic Software.-Products:...	8.6
Testing Anywhere Testing Anywhere Testing Anywhere is software produced by San Jose-based Automation Anywhere, Inc. . The software allows testers and developers to test applications, Web sites, objects, controls and GUI front-ends.The company has been featured in SD Times...	Automation Anywhere	7.0
TestPartner TestPartner TestPartner is a GUI software testing tool from Micro Focus that is intended to enable software development project teams to functionally automate and test application Graphical User Interfaces, with the goal of being able to accomplish more application testing in a given amount of time than could...	Micro Focus	6.3
TOSCA Testsuite Tosca (Software) TOSCA Testsuite is a software tool for the automated execution of functional and regression software testing. In addition to test automation functions, TOSCA includes integrated test management, a graphical user interface , a command line interface and an application programming interface...	TRICENTIS Technology & Consulting	7.3.1
Visual Studio Test Professional Visual Studio Test Professional Visual Studio Test Professional is an integrated testing toolset developed by Microsoft that delivers a complete plan-test-track workflow for in-context collaboration between testers and developers, in order to increase testers’ visibility to the overall project.-Testing Tools:* Manual...	Microsoft Microsoft Microsoft Corporation is an American public multinational corporation headquartered in Redmond, Washington, USA that develops, manufactures, licenses, and supports a wide range of products and services predominantly related to computing through its various product divisions...	2010
WATIR Watir Web Application Testing in Ruby is a toolkit used to automate browser-based tests during web application development. This automated test tool uses the Ruby scripting language to drive Internet Explorer, Mozilla Firefox, Google Chrome, Opera and Safari, and is available as a RubyGems gem...	Open source	1.6.5
WebUI Test Studio WebUI Test Studio Test Studio is a tool for automated testing of web applications developed by Telerik. - Overview :Test Studio is used for testing all type of web applications without the need of writing code. The tool comes in two editions – a Visual Studio plug-in for software developers, and a standalone...	Telerik, Inc.	2011.2

See also

• List of GUI testing tools
• Software testing
  Software testing
  Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
  
  
• System testing
  System testing
  System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or...
  
  
• Test automation framework
  Test automation framework
  A test automation framework is a set of assumptions, concepts and tools that provide support for automated software testing. The main advantage of such a framework is the low cost for maintenance. If there is change to any test case then only the test case file needs to be updated and the Driver...
  
  
• Unit test
  Unit test
  In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use.A unit is the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function...
  
  

External links

• Automation Myths by M. N. Alam
• Generating Test Cases Automatically
• Practical Experience in Automated Testing
• Test Automation: Delivering Business Value
• Test Automation Snake Oil by James Bach
• When Should a Test Be Automated? by Brian Marick
• Why Automation Projects Fail by Art Beall
• Guidelines for Test Automation framework
• Advanced Test Automation
• Seven Steps to Test Automation Success