Exploratory test
Encyclopedia
Exploratory testing is an approach to software testing
that is concisely described as simultaneous learning, test design
and test execution. Cem Kaner
, who coined the term in 1983, now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."
While the software is being tested, the tester learns things that together with experience and creativity
generates new good tests to run. Exploratory testing is often thought of as a black box testing
technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.
was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the Context-Driven School) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by Cem Kaner
in his book Testing Computer Software and expanded upon in Lessons Learned in Software Testing. Exploratory testing can be as disciplined as any other intellectual activity.
s and finding defects
. The more the tester knows about the product and different test method
s, the better the testing will be.
To further explain, comparison can be made of freestyle exploratory testing to its antithesis scripted testing
. In this activity test cases are designed in advance. This includes both the individual steps and the expected results. These tests are later performed by a tester who compares the actual result with the expected. When performing exploratory testing, expectations are open. Some results may be predicted and expected; others may not. The tester configures, operates, observes, and evaluates the product and its behaviour, critically investigating the result, and reporting information that seems likely to be a bug (which threatens the value of the product to some person) or an issue (which threatens the quality of the testing effort).
In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards either one, depending on context.
According to Cem Kaner
& James Bach, exploratory testing is more a mindset
or "...a way of thinking about testing" than a methodology. They also say that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).
The documentation of exploratory testing ranges from documenting all tests performed to just documenting the bugs
. During pair testing
, two persons create test cases together; one performs them, and the other documents.
Session-based test
ing is a method specifically designed to make exploratory testing auditable and measurable on a wider scale.
Exploratory testers often use tools, including screen capture or video tools as a record of the exploratory session, or tools to quickly help generate situations of interest, e.g. James Bach's Perlclip.
Another major benefit is that testers can use deductive reasoning
based on the results of previous results to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently.
Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."
Disadvantages are that tests invented and performed on the fly can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.
Freestyle exploratory test ideas, when revisited, are unlikely to be performed in exactly the same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to repeat specific details of the earlier tests. This can be controlled with specific instruction to the tester, or by preparing automated tests where feasible, appropriate, and necessary, and ideally as close to the unit level as possible.
The approach can also be used to verify that previous testing has found the most important defects.
Software testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
that is concisely described as simultaneous learning, test design
Test design
In software engineering, test design is the act of creating and writing test suites for testing a software.- Definition :Test design could require all or one of:* knowledge of the software, and the business area it operates on,...
and test execution. Cem Kaner
Cem Kaner
Cem Kaner J.D., Ph.D., is a Professor of Software Engineering at Florida Institute of Technology, and the Director of Florida Tech's Center for Software Testing Education & Research since 2004...
, who coined the term in 1983, now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."
While the software is being tested, the tester learns things that together with experience and creativity
Creativity
Creativity refers to the phenomenon whereby a person creates something new that has some kind of value. What counts as "new" may be in reference to the individual creator, or to the society or domain within which the novelty occurs...
generates new good tests to run. Exploratory testing is often thought of as a black box testing
Black box testing
Black-box testing is a method of software testing that tests the functionality of an application as opposed to its internal structures or workings . Specific knowledge of the application's code/internal structure and programming knowledge in general is not required...
technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.
History
Exploratory testing has always been performed by skilled testers. In the early 1990s, ad hocAd hoc
Ad hoc is a Latin phrase meaning "for this". It generally signifies a solution designed for a specific problem or task, non-generalizable, and not intended to be able to be adapted to other purposes. Compare A priori....
was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the Context-Driven School) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by Cem Kaner
Cem Kaner
Cem Kaner J.D., Ph.D., is a Professor of Software Engineering at Florida Institute of Technology, and the Director of Florida Tech's Center for Software Testing Education & Research since 2004...
in his book Testing Computer Software and expanded upon in Lessons Learned in Software Testing. Exploratory testing can be as disciplined as any other intellectual activity.
Description
Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester's skill of inventing test caseTest case
A test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
s and finding defects
Software bug
A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program's...
. The more the tester knows about the product and different test method
Test method
A test method is a definitive procedure that produces a test result.A test can be considered as technical operation that consists of determination of one or more characteristics of a given product, process or service according to a specified procedure. Often a test is part of an experiment.The test...
s, the better the testing will be.
To further explain, comparison can be made of freestyle exploratory testing to its antithesis scripted testing
Test script
A test script in software testing is a set of instructions that will be performed on the system under test to test that the system functions as expected.There are various means for executing test scripts.* Manual testing...
. In this activity test cases are designed in advance. This includes both the individual steps and the expected results. These tests are later performed by a tester who compares the actual result with the expected. When performing exploratory testing, expectations are open. Some results may be predicted and expected; others may not. The tester configures, operates, observes, and evaluates the product and its behaviour, critically investigating the result, and reporting information that seems likely to be a bug (which threatens the value of the product to some person) or an issue (which threatens the quality of the testing effort).
In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards either one, depending on context.
According to Cem Kaner
Cem Kaner
Cem Kaner J.D., Ph.D., is a Professor of Software Engineering at Florida Institute of Technology, and the Director of Florida Tech's Center for Software Testing Education & Research since 2004...
& James Bach, exploratory testing is more a mindset
Mindset
In decision theory and general systems theory, a mindset is a set of assumptions, methods or notations held by one or more people or groups of people which is so established that it creates a powerful incentive within these people or groups to continue to adopt or accept prior behaviors, choices,...
or "...a way of thinking about testing" than a methodology. They also say that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).
The documentation of exploratory testing ranges from documenting all tests performed to just documenting the bugs
Software bug
A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program's...
. During pair testing
Pair Testing
Pair testing is a software development technique in which two team members work together at one keyboard to test the software application. One does the testing and the other analyzes or reviews the testing...
, two persons create test cases together; one performs them, and the other documents.
Session-based test
Session-based test
Session-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with Scenario testing...
ing is a method specifically designed to make exploratory testing auditable and measurable on a wider scale.
Exploratory testers often use tools, including screen capture or video tools as a record of the exploratory session, or tools to quickly help generate situations of interest, e.g. James Bach's Perlclip.
Benefits and drawbacks
The main advantage of exploratory testing is that less preparation is needed, important bugs are found quickly, and at execution time, the approach tends to be more intellectually stimulating than execution of scripted tests.Another major benefit is that testers can use deductive reasoning
Deductive reasoning
Deductive reasoning, also called deductive logic, is reasoning which constructs or evaluates deductive arguments. Deductive arguments are attempts to show that a conclusion necessarily follows from a set of premises or hypothesis...
based on the results of previous results to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently.
Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."
Disadvantages are that tests invented and performed on the fly can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.
Freestyle exploratory test ideas, when revisited, are unlikely to be performed in exactly the same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to repeat specific details of the earlier tests. This can be controlled with specific instruction to the tester, or by preparing automated tests where feasible, appropriate, and necessary, and ideally as close to the unit level as possible.
Usage
Exploratory testing is particularly suitable if requirements and specifications are incomplete, or if there is lack of time.The approach can also be used to verify that previous testing has found the most important defects.
See also
- Ad hoc testingAd hoc testingAd hoc testing is a commonly used term for software testing performed without planning and documentation ....
- Context-Driven SchoolSoftware testing controversiesThere is considerable variety among software testing writers and consultants about what constitutes responsible software testing. Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester...
- San Francisco depotSan Francisco depotSan Francisco depot is a mnemonic for the SFDPO software exploratory testing heuristic. SFDPO stands for Structure, Function, Data, Platform and Operations. Each of these represents a different aspect of a software product.- Structure :...
- Session-based testSession-based testSession-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with Scenario testing...
ing - Software testingSoftware testingSoftware testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
- Test caseTest caseA test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
- Test scriptTest scriptA test script in software testing is a set of instructions that will be performed on the system under test to test that the system functions as expected.There are various means for executing test scripts.* Manual testing...
External links
- James Bach, Exploratory Testing Explained
- Cem Kaner, James Bach, The Nature of Exploratory Testing, 2004
- Cem Kaner, James Bach, The Seven Basic Principles of the Context-Driven School
- Jonathan Kohl, Exploratory Testing: Finding the Music of Software Investigation, Kohl Concepts Inc., 2007
- Chris Agruss, Bob Johnson, Ad Hoc Software Testing