Software testing
Encyclopedia
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include, but are not limited to, the process of executing a program or application with the intent of finding software bug
s (errors or other defects).
Software testing can be stated as the process of validating and verifying that a software program/application/product:
Software testing, depending on the testing method employed, can be implemented at any time in the development process. However, most of the test effort occurs after the requirements have been defined and the coding process has been completed. As such, the methodology of the test is governed by the software development methodology adopted.
Different software development models will focus the test effort at different points in the development process. Newer development models, such as Agile
, often employ test driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers. In a more traditional model, most of the test execution occurs after the requirements have been defined and the coding process has been completed.
, comparable products, past versions of the same product, inferences about intended or expected purpose, user or customer expectations, relevant standards, applicable laws, or other criteria.
Every software product has a target audience. For example, the audience for video game software is completely different from banking software. Therefore, when an organization develops or otherwise invests in a software product, it can assess whether the software product will be acceptable to its end users, its target audience, its purchasers, and other stakeholders. Software testing is the process of attempting to make this assessment.
A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed.
and William C. Hetzel
classified in 1988 the phases and goals in software testing in the following stages:
Non-functional testing refers to aspects of the software that may not be related to a specific function or user action, such as scalability
or other performance
, behavior under certain constraints, or security
. Non-functional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users.
such as testability
, scalability
, maintainability
, usability
, performance
, and security
.
Software faults occur through the following processes. A programmer makes an error
(mistake), which results in a defect
(fault, bug) in the software source code
. If this defect is executed, in certain situations the system will produce wrong results, causing a failure
. Not all defects will necessarily result in failures. For example, defects in dead code
will never result in failures. A defect can turn into a failure when the environment is changed. Examples of these changes in environment include the software being run on a new hardware
platform, alterations in source data
or interacting with different software. A single defect may result in a wide range of failure symptoms.
The following table shows the cost of fixing the defect depending on the stage it was found. For example, if a problem in the requirements is found only post-release, then it would cost 10–100 times more to fix than if it had already been found by the requirements review. Modern continuous deployment practices, and cloud-based services may cost less for re-deployment and maintenance than in the past.
with other application software
, operating system
s (or operating system versions, old or new), or target environments that differ greatly from the original (such as a terminal
or GUI
application intended to be run on the desktop
now being required to become a web application
, which must render in a web browser
). For example, in the case of a lack of backward compatibility
, this can occur because the programmers develop and test software only on the latest version of the target environment, which not all users may be running. This results in the unintended consequence that the latest work may not function on earlier versions of the target environment, or on older hardware that earlier versions of the target environment was capable of using. Sometimes such issues can be fixed by proactively abstracting
operating system functionality into a separate program module
or library.
s in a software product can be very large and defects that occur infrequently are difficult to find in testing. More significantly, non-functional
dimensions of quality (how it is supposed to be versus what it is supposed to do)—usability
, scalability
, performance
, compatibility
, reliability—can be highly subjective; something that constitutes sufficient value to one person may be intolerable to another.
, walkthrough
s, or inspections
are considered as static testing
, whereas actually executing programmed code with a given set of test case
s is referred to as dynamic testing
. Static testing can be (and unfortunately in practice often is) omitted. Dynamic testing takes place when the program itself is used for the first time (which is generally considered the beginning of the testing stage). Dynamic testing may begin before the program is 100% complete in order to test particular sections of code (modules or discrete functions). Typical techniques for this are either using stubs
/drivers or execution from a debugger
environment. For example, spreadsheet
programs are, by their very nature, tested to a large extent interactively ("on the fly
"), with results displayed immediately after each calculation or text manipulation.
The terms verification and validation are commonly used interchangeably in the industry; it is also common to see these two terms incorrectly defined. According to the IEEE Standard Glossary of Software Engineering Terminology:
According to the IS0 9000 standard:
(SQA) process. In SQA, software process specialists and auditors are concerned for the software development process rather than just the artifacts such as documentation, code and systems. They examine and change the software engineering process itself to reduce the amount of faults that end up in the delivered software: the so-called defect rate.
What constitutes an "acceptable defect rate" depends on the nature of the software; A flight simulator video game would have much higher defect tolerance than software for an actual airplane.
Although there are close links with SQA, testing departments often exist independently, and there may be no SQA function in some companies.
Software testing is a task intended to detect defects in software by contrasting a computer program's expected results with its actual results for a given set of inputs. By contrast, QA (quality assurance) is the implementation of policies and procedures intended to prevent defects from occurring in the first place.
Types of white box testing
Test coverage
They both return a code coverage
metric
, measured as a percentage
.
, boundary value analysis
, all-pairs testing
, fuzz testing
, model-based testing
, exploratory testing and specification-based testing.
Therefore, black box testing has the advantage of "an unaffiliated opinion", on the one hand, and the disadvantage of "blind exploring", on the other.
between two modules of code written by two different developers, where only the interfaces are exposed for test. However, modifying a data repository does qualify as grey box, as the user would not normally be able to change the data outside of the system under test. Grey box testing may also include reverse engineering to determine, for instance, boundary values or error messages.
By knowing the underlying concepts of how the software works, the tester makes better-informed testing choices while testing the software from outside. Typically, a grey box tester will be permitted to set up his testing environment; for instance, seeding a database
; and the tester can observe the state of the product being tested after performing certain actions. For instance, he/she may fire an SQL
query on the database and then observe the database, to ensure that the expected changes have been reflected. Grey box testings implements intelligent test scenarios, based on limited information. This will particularly apply to data type handling, exception handling, and so on.
At the core of visual testing is the idea that showing someone a problem (or a test failure), rather than just describing it, greatly increases clarity and understanding. Visual testing therefore requires the recording of the entire test process – capturing everything that occurs on the test system in video format. Output videos are supplemented by real-time tester input via picture-in-a-picture webcam and audio commentary from microphones.
Visual testing provides a number of advantages. The quality of communication is increased dramatically because testers can show the problem (and the events leading up to it) to the developer as opposed to just describing it and the need to replicate test failures will cease to exist in many cases. The developer will have all the evidence he requires of a test failure and can instead focus on the cause of the fault and how it should be fixed.
Visual testing is particularly well-suited for environments that deploy agile methods in their development of software, since agile methods require greater communication between testers and developers and collaboration within small teams.
Ad hoc testing
and exploratory testing are important methodologies for checking software integrity, because they require less preparation time to implement, whilst important bugs can be found quickly. In ad hoc testing, where testing takes place in an improvised, impromptu way, the ability of a test tool to visually record everything that occurs on a system becomes very important.
Visual testing is gathering recognition in customer acceptance and usability testing
, because the test can be used by many individuals involved in the development process. For the customer, it becomes easy to provide detailed bug reports and feedback, and for program users, visual testing can record user actions on screen, as well as their voice and image, to provide a complete picture at the time of software failure for the developer.
The main levels during the development process as defined by the SWEBOK guide are unit-, integration-, and system testing that are distinguished by the test target without implying a specific process model. Other test levels are classified by the testing objective.
These types of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. One function might have multiple tests, to catch corner case
s or other branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to assure that the building blocks the software uses work independently of each other.
Integration testing
works to expose defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a system.
tests a completely integrated system to verify that it meets its requirements.
verifies that a system is integrated to any external or third-party systems defined in the system requirements.
s, or old bugs that have come back. Such regressions occur whenever software functionality that was previously working correctly stops working as intended. Typically, regressions occur as an unintended consequence
of program changes, when the newly developed part of the software collides with the previously existing code. Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged. The depth of testing depends on the phase in the release process and the risk
of the added features. They can either be complete, for changes added late in the release or deemed to be risky, to very shallow, consisting of positive tests on each feature, if the changes are early in the release or deemed to be of low risk.
, in the form of fuzzing
, is an example of non-functional testing. Non-functional testing, especially for software, is designed to establish whether the device under test can tolerate invalid or unexpected inputs, thereby establishing the robustness of input validation routines as well as error-management routines. Various commercial non-functional testing tools are linked from the software fault injection
page; there are also numerous open-source and free software tools available that perform non-functional testing.
Load testing is primarily concerned with testing that can continue to operate under a specific load, whether that be large quantities of data or a large number of users
. This is generally referred to as software scalability
. The related load testing activity of when performed as a non-functional activity is often referred to as endurance testing. Volume testing
is a way to test functionality. Stress testing
is a way to test reliability under unexpected or rare workloads. Stability testing (often referred to as load or endurance testing) checks to see if the software can continuously function well in or above an acceptable period.
There is little agreement on what the specific goals of performance testing are. The terms load testing, performance testing, reliability testing, and volume testing, are often used interchangeably.
is needed to check if the user interface is easy to use and understand. It is concerned mainly with the use of the application.
is essential for software that processes confidential data to prevent system intrusion by hackers
.
can be automatically tested without actual translation, by using pseudolocalization
. It will verify that the application still works, even after it has been translated into a new language or adapted for a new culture (such as different currencies or time zones).
Actual translation to human languages must be tested, too. Possible localization failures include:
To avoid these and other localization problems, a tester who knows the target language must run the program with all the possible use cases for translation to see if the messages are readable, translated correctly in context and don't cause failures.
buffer to compensate for project delays, thereby compromising the time devoted to testing.
Another practice is to start software testing at the same moment the project starts and it is a continuous process until the project finishes.
and the agile software development
movement, adhere to a "test-driven software development
" model. In this process, unit tests are written first, by the software engineers
(often with pair programming
in the extreme programming methodology). Of course these tests fail initially; as they are expected to. Then as code is written it passes incrementally larger portions of the test suites. The test suites are continuously updated as new failure conditions and corner cases are discovered, and they are integrated with any regression tests that are developed. Unit tests are maintained along with the rest of the software source code and generally integrated into the build process (with inherently interactive tests being relegated to a partially manual build acceptance process). The ultimate goal of this test process is to achieve continuous deployment where software updates can be published to the public frequently.
, especially groups that use
test-driven development
. There are many frameworks to write tests in, and continuous integration
software will run tests automatically every time code is checked into a version control system.
While automation cannot reproduce everything that a human can do (and all the ways they think of doing it), it can be very useful for regression testing. However, it does require a well-developed test suite
of testing scripts in order to be truly useful.
s.
Testing/debug tools include features such as:
Some of these features may be incorporated into an Integrated Development Environment
(IDE).
, but can also include more technical requirements as described under the ISO
standard ISO/IEC 9126, such as capability, reliability
, efficiency
, portability
, maintainability
, compatibility, and usability
.
There are a number of frequently-used software measures, often called metrics, which are used to assist in determining the state of the software or the adequacy of the testing.
.
Test plan: A test specification is called a test plan
. The developers are well aware what test plans will be executed and this information is made available to management and the developers. The idea is to make them more cautious when developing their code or making additional changes. Some companies have a higher-level document called a test strategy
.
Traceability matrix: A traceability matrix
is a table that correlates requirements or design documents to test documents. It is used to change tests when related source documents are changed, to select test cases for execution when planning for regression tests by considering requirement coverage.
Test case: A test case
normally consists of a unique identifier, requirement references from a design specification, preconditions, events, a series of steps (also known as actions) to follow, input, output, expected result, and actual result. Clinically defined a test case is an input and an expected result. This can be as pragmatic as 'for condition x your derived result is y', whereas other test cases described in more detail the input scenario and what results might be expected. It can occasionally be a series of steps (but often steps are contained in a separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one expected result or expected outcome. The optional fields are a test case ID, test step, or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database, or other common repository. In a database system, you may also be able to see past test results, who generated the results, and what system configuration was used to generate those results. These past results would usually be stored in a separate table.
Test script: A test script
is a procedure, or programing code that replicates user actions. Initially the term was derived from the product of work created by automated regression test tools. Test Case will be a baseline to create test scripts using a tool or a program.
Test suite: The most common term for a collection of test cases is a test suite
. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.
Test data: In most cases, multiple sets of values or data are used to test the same functionality of a particular feature. All the test values and changeable environmental components are collected in separate files and stored as test data. It is also useful to provide this data to the client and with the product or a project.
Test harness: The software, tools, samples of data input and output, and configurations are all referred to collectively as a test harness
.
Software testing certification types
Testing certifications
Quality assurance certifications
include:
What constitutes responsible software testing? : Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester to select or invent testing practices to suit each unique situation.
Agile vs. traditional : Should testers learn to work under conditions of uncertainty and constant change or should they aim at process "maturity"
? The agile testing
movement has received growing popularity since 2006 mainly in commercial circles, whereas government and military software providers use this methodology but also the traditional test-last models (e.g. in the Waterfall model
).
Exploratory test vs. scripted : Should tests be designed at the same time as they are executed or should they be designed beforehand?
Manual testing vs. automated : Some writers believe that test automation
is so expensive relative to its value that it should be used sparingly. More in particular, test-driven development
states that developers should write unit-tests of the XUnit
type before coding the functionality. The tests then can be considered as a way to capture and implement the requirements.
Software design vs. software implementation : Should testing be carried out only at the end or throughout the whole process?
Who watches the watchmen? : The idea is that any form of observation is also an interaction—the act of testing can also affect that which is being tested.
Software bug
A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program's...
s (errors or other defects).
Software testing can be stated as the process of validating and verifying that a software program/application/product:
- meets the requirements that guided its design and development;
- works as expected; and
- can be implemented with the same characteristics.
Software testing, depending on the testing method employed, can be implemented at any time in the development process. However, most of the test effort occurs after the requirements have been defined and the coding process has been completed. As such, the methodology of the test is governed by the software development methodology adopted.
Different software development models will focus the test effort at different points in the development process. Newer development models, such as Agile
Agile software development
Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams...
, often employ test driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers. In a more traditional model, most of the test execution occurs after the requirements have been defined and the coding process has been completed.
Overview
Testing can never completely identify all the defects within software. Instead, it furnishes a criticism or comparison that compares the state and behavior of the product against oracles—principles or mechanisms by which someone might recognize a problem. These oracles may include (but are not limited to) specifications, contractsDesign by contract
Design by contract , also known as programming by contract and design-by-contract programming, is an approach to designing computer software...
, comparable products, past versions of the same product, inferences about intended or expected purpose, user or customer expectations, relevant standards, applicable laws, or other criteria.
Every software product has a target audience. For example, the audience for video game software is completely different from banking software. Therefore, when an organization develops or otherwise invests in a software product, it can assess whether the software product will be acceptable to its end users, its target audience, its purchasers, and other stakeholders. Software testing is the process of attempting to make this assessment.
A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed.
History
The separation of debugging from testing was initially introduced by Glenford J. Myers in 1979. Although his attention was on breakage testing ("a successful test is one that finds a bug") it illustrated the desire of the software engineering community to separate fundamental development activities, such as debugging, from that of verification. Dave GelperinDave Gelperin
Doctor David Gelperin chaired the working groups developing the IEEE 829-1989 software testing documentation standard. With Jerry E. Durant he went on to develop High Impact Inspection Technology that builds upon traditional inspections but utilizes a test driven additive.Gelperin received his PhD...
and William C. Hetzel
William C. Hetzel
Doctor William C. Hetzel wrote the highly-cited 1991 book The Complete Guide to Software Testing, which continues to be pivotal in the software testing culture and remains a consistent source of reference. In 1972 he was the organiser of the first conference dedicated to software testing, "Software...
classified in 1988 the phases and goals in software testing in the following stages:
- Until 1956 - Debugging oriented
- 1957–1978 - Demonstration oriented
- 1979–1982 - Destruction oriented
- 1983–1987 - Evaluation oriented
- 1988–2000 - Prevention oriented
Scope
A primary purpose of testing is to detect software failures so that defects may be discovered and corrected. Testing cannot establish that a product functions properly under all conditions but can only establish that it does not function properly under specific conditions. The scope of software testing often includes examination of code as well as execution of that code in various environments and conditions as well as examining the aspects of code: does it do what it is supposed to do and do what it needs to do. In the current culture of software development, a testing organization may be separate from the development team. There are various roles for testing team members. Information derived from software testing may be used to correct the process by which software is developed.Functional vs non-functional testing
Functional testing refers to activities that verify a specific action or function of the code. These are usually found in the code requirements documentation, although some development methodologies work from use cases or user stories. Functional tests tend to answer the question of "can the user do this" or "does this particular feature work."Non-functional testing refers to aspects of the software that may not be related to a specific function or user action, such as scalability
Scalability
In electronics scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth...
or other performance
Performance
A performance, in performing arts, generally comprises an event in which a performer or group of performers behave in a particular way for another group of people, the audience. Choral music and ballet are examples. Usually the performers participate in rehearsals beforehand. Afterwards audience...
, behavior under certain constraints, or security
Computer security
Computer security is a branch of computer technology known as information security as applied to computers and networks. The objective of computer security includes protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to...
. Non-functional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users.
Defects and failures
Not all software defects are caused by coding errors. One common source of expensive defects is caused by requirement gaps, e.g., unrecognized requirements, that result in errors of omission by the program designer. A common source of requirements gaps is non-functional requirementsNon-functional requirements
In systems engineering and requirements engineering, a non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. This should be contrasted with functional requirements that define specific behavior or...
such as testability
Software testability
Software testability is the degree to which a software artifact supports testing in a given test context....
, scalability
Scalability
In electronics scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth...
, maintainability
Maintainability
In engineering, maintainability is the ease with which a product can be maintained in order to:* isolate defects or their cause* correct defects or their cause* meet new requirements* make future maintenance easier, or* cope with a changed environment...
, usability
Usability
Usability is the ease of use and learnability of a human-made object. The object of use can be a software application, website, book, tool, machine, process, or anything a human interacts with. A usability study may be conducted as a primary job function by a usability analyst or as a secondary job...
, performance
Computer performance
Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used.Depending on the context, good computer performance may involve one or more of the following:...
, and security
Computer security
Computer security is a branch of computer technology known as information security as applied to computers and networks. The objective of computer security includes protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to...
.
Software faults occur through the following processes. A programmer makes an error
Human Error
Human Error is the stage name of Rafał Kuczynski , a polish electronic musician, working mostly in the ambient music genre, produced only with a computer...
(mistake), which results in a defect
Fault (technology)
In document ISO/CD 10303-226, a fault is defined as an abnormal condition or defect at the component, equipment, or sub-system level which may lead to a failure....
(fault, bug) in the software source code
Source code
In computer science, source code is text written using the format and syntax of the programming language that it is being written in. Such a language is specially designed to facilitate the work of computer programmers, who specify the actions to be performed by a computer mostly by writing source...
. If this defect is executed, in certain situations the system will produce wrong results, causing a failure
Failure
Failure refers to the state or condition of not meeting a desirable or intended objective, and may be viewed as the opposite of success. Product failure ranges from failure to sell the product to fracture of the product, in the worst cases leading to personal injury, the province of forensic...
. Not all defects will necessarily result in failures. For example, defects in dead code
Dead code
Dead code is a computer programming term for code in the source code of a program which is executed but whose result is never used in any other computation...
will never result in failures. A defect can turn into a failure when the environment is changed. Examples of these changes in environment include the software being run on a new hardware
Hardware
Hardware is a general term for equipment such as keys, locks, hinges, latches, handles, wire, chains, plumbing supplies, tools, utensils, cutlery and machine parts. Household hardware is typically sold in hardware stores....
platform, alterations in source data
Source data
Source data is the origin of information found in electronic media.Often when data is captured in one electronic system and then transferred to another, there is a loss of audit trail or the inherent data cannot be absolutely verified. There are systems that provide for absolute data export but...
or interacting with different software. A single defect may result in a wide range of failure symptoms.
Finding faults early
It is commonly believed that the earlier a defect is found the cheaper it is to fix it.The following table shows the cost of fixing the defect depending on the stage it was found. For example, if a problem in the requirements is found only post-release, then it would cost 10–100 times more to fix than if it had already been found by the requirements review. Modern continuous deployment practices, and cloud-based services may cost less for re-deployment and maintenance than in the past.
| Cost to fix a defect | Time detected | |||||
|---|---|---|---|---|---|---|
| Requirements | Architecture | Construction | System test | Post-release | ||
| Time introduced | Requirements | 1× | 3× | 5–10× | 10× | 10–100× |
| Architecture | - | 1× | 10× | 15× | 25–100× | |
| Construction | - | - | 1× | 10× | 10–25× | |
Compatibility
A common cause of software failure (real or perceived) is a lack of compatibilityComputer compatibility
A family of computer models is said to be compatible if certain software that runs on one of the models can also be run on all other models of the family. The computer models may differ in performance, reliability or some other characteristic...
with other application software
Application software
Application software, also known as an application or an "app", is computer software designed to help the user to perform specific tasks. Examples include enterprise software, accounting software, office suites, graphics software and media players. Many application programs deal principally with...
, operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
s (or operating system versions, old or new), or target environments that differ greatly from the original (such as a terminal
Computer terminal
A computer terminal is an electronic or electromechanical hardware device that is used for entering data into, and displaying data from, a computer or a computing system...
or GUI
Gui
Gui or guee is a generic term to refer to grilled dishes in Korean cuisine. These most commonly have meat or fish as their primary ingredient, but may in some cases also comprise grilled vegetables or other vegetarian ingredients. The term derives from the verb, "gupda" in Korean, which literally...
application intended to be run on the desktop
Desktop metaphor
The desktop metaphor is an interface metaphor which is a set of unifying concepts used by graphical user interfaces to help users more easily interact with the computer. The desktop metaphor treats the monitor of a computer as if it is the user's desktop, upon which objects such as documents and...
now being required to become a web application
Web application
A web application is an application that is accessed over a network such as the Internet or an intranet. The term may also mean a computer software application that is coded in a browser-supported language and reliant on a common web browser to render the application executable.Web applications are...
, which must render in a web browser
Web browser
A web browser is a software application for retrieving, presenting, and traversing information resources on the World Wide Web. An information resource is identified by a Uniform Resource Identifier and may be a web page, image, video, or other piece of content...
). For example, in the case of a lack of backward compatibility
Backward compatibility
In the context of telecommunications and computing, a device or technology is said to be backward or downward compatible if it can work with input generated by an older device...
, this can occur because the programmers develop and test software only on the latest version of the target environment, which not all users may be running. This results in the unintended consequence that the latest work may not function on earlier versions of the target environment, or on older hardware that earlier versions of the target environment was capable of using. Sometimes such issues can be fixed by proactively abstracting
Abstraction (computer science)
In computer science, abstraction is the process by which data and programs are defined with a representation similar to its pictorial meaning as rooted in the more complex realm of human life and language with their higher need of summarization and categorization , while hiding away the...
operating system functionality into a separate program module
Modular programming
Modular programming is a software design technique that increases the extent to which software is composed of separate, interchangeable components called modules by breaking down program functions into modules, each of which accomplishes one function and contains everything necessary to accomplish...
or library.
Input combinations and preconditions
A very fundamental problem with software testing is that testing under all combinations of inputs and preconditions (initial state) is not feasible, even with a simple product. This means that the number of defectSoftware bug
A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program's...
s in a software product can be very large and defects that occur infrequently are difficult to find in testing. More significantly, non-functional
Non-functional requirements
In systems engineering and requirements engineering, a non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. This should be contrasted with functional requirements that define specific behavior or...
dimensions of quality (how it is supposed to be versus what it is supposed to do)—usability
Usability
Usability is the ease of use and learnability of a human-made object. The object of use can be a software application, website, book, tool, machine, process, or anything a human interacts with. A usability study may be conducted as a primary job function by a usability analyst or as a secondary job...
, scalability
Scalability
In electronics scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth...
, performance
Computer performance
Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used.Depending on the context, good computer performance may involve one or more of the following:...
, compatibility
Backward compatibility
In the context of telecommunications and computing, a device or technology is said to be backward or downward compatible if it can work with input generated by an older device...
, reliability—can be highly subjective; something that constitutes sufficient value to one person may be intolerable to another.
Static vs. dynamic testing
There are many approaches to software testing. ReviewsCode review
Code review is systematic examination of computer source code. It is intended to find and fix mistakes overlooked in the initial development phase, improving both the overall quality of software and the developers' skills...
, walkthrough
Software walkthrough
In software engineering, a walkthrough or walk-through is a form of software peer review "in which a designer or programmer leads members of the development team and other interested parties through a software product, and the participants ask questions and make comments about possible errors,...
s, or inspections
Software inspection
Inspection in software engineering, refers to peer review of any work product by trained individuals who look for defects using a well defined process...
are considered as static testing
Static testing
Static testing is a form of software testing where the software isn't actually used. This is in contrast to dynamic testing. It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or document. It is primarily syntax checking of the code and/or manually...
, whereas actually executing programmed code with a given set of test case
Test case
A test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
s is referred to as dynamic testing
Dynamic testing
Dynamic testing is a term used in software engineering to describe the testing of the dynamic behavior of code. That is, dynamic analysis refers to the examination of the physical response from the system to variables that are not constant and change with time...
. Static testing can be (and unfortunately in practice often is) omitted. Dynamic testing takes place when the program itself is used for the first time (which is generally considered the beginning of the testing stage). Dynamic testing may begin before the program is 100% complete in order to test particular sections of code (modules or discrete functions). Typical techniques for this are either using stubs
Method stub
A method stub or simply stub in software development is a piece of code used to stand in for some other programming functionality. A stub may simulate the behavior of existing code or be a temporary substitute for yet-to-be-developed code...
/drivers or execution from a debugger
Debugger
A debugger or debugging tool is a computer program that is used to test and debug other programs . The code to be examined might alternatively be running on an instruction set simulator , a technique that allows great power in its ability to halt when specific conditions are encountered but which...
environment. For example, spreadsheet
Spreadsheet
A spreadsheet is a computer application that simulates a paper accounting worksheet. It displays multiple cells usually in a two-dimensional matrix or grid consisting of rows and columns. Each cell contains alphanumeric text, numeric values or formulas...
programs are, by their very nature, tested to a large extent interactively ("on the fly
On the fly
-Colloquial usage:In colloquial use, on the fly means something created when needed. The phrase is used to mean:# something that was not planned ahead# changes that are made during the execution of same activity: ex tempore, impromptu.-Automotive usage:...
"), with results displayed immediately after each calculation or text manipulation.
Software verification and validation
Software testing is used in association with verification and validation:- Verification: Have we built the software right? (i.e., does it match the specification).
- Validation: Have we built the right software? (i.e., is this what the customer wants).
The terms verification and validation are commonly used interchangeably in the industry; it is also common to see these two terms incorrectly defined. According to the IEEE Standard Glossary of Software Engineering Terminology:
- Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase.
- Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.
According to the IS0 9000 standard:
- Verification is confirmation by examination and through provision of objective evidence that specified requirements have been fulfilled.
- Validation is confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled.
The software testing team
Software testing can be done by software testers. Until the 1980s the term "software tester" was used generally, but later it was also seen as a separate profession. Regarding the periods and the different goals in software testing, different roles have been established: manager, test lead, test designer, tester, automation developer, and test administrator.Software quality assurance (SQA)
Though controversial, software testing is a part of the software quality assuranceSoftware quality assurance
Software quality assurance consists of a means of monitoring the software engineering processes and methods used to ensure quality. The methods by which this is accomplished are many and varied, and may include ensuring conformance to one or more standards, such as ISO 9000 or a model such as...
(SQA) process. In SQA, software process specialists and auditors are concerned for the software development process rather than just the artifacts such as documentation, code and systems. They examine and change the software engineering process itself to reduce the amount of faults that end up in the delivered software: the so-called defect rate.
What constitutes an "acceptable defect rate" depends on the nature of the software; A flight simulator video game would have much higher defect tolerance than software for an actual airplane.
Although there are close links with SQA, testing departments often exist independently, and there may be no SQA function in some companies.
Software testing is a task intended to detect defects in software by contrasting a computer program's expected results with its actual results for a given set of inputs. By contrast, QA (quality assurance) is the implementation of policies and procedures intended to prevent defects from occurring in the first place.
The box approach
Software testing methods are traditionally divided into white- and black-box testing. These two approaches are used to describe the point of view that a test engineer takes when designing test cases.White box testing
White box testing is when the tester has access to the internal data structures and algorithms including the code that implement these.Types of white box testing
- The following types of white box testing exist:
- APIApplication programming interfaceAn application programming interface is a source code based specification intended to be used as an interface by software components to communicate with each other...
testing (application programming interface) - testing of the application using public and private APIs - Code coverageCode coverageCode coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing....
- creating tests to satisfy some criteria of code coverage (e.g., the test designer can create tests to cause all statements in the program to be executed at least once) - Fault injectionFault injectionIn software testing, fault injection is a technique for improving the coverage of a test by introducing faults to test code paths, in particular error handling code paths, that might otherwise rarely be followed. It is often used with stress testing and is widely considered to be an important part...
methods - improving the coverage of a test by introducing faults to test code paths - Mutation testing methods
- Static testingStatic testingStatic testing is a form of software testing where the software isn't actually used. This is in contrast to dynamic testing. It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or document. It is primarily syntax checking of the code and/or manually...
- White box testing includes all static testing
- API
Test coverage
- White box testing methods can also be used to evaluate the completeness of a test suite that was created with black box testing methods. This allows the software team to examine parts of a system that are rarely tested and ensures that the most important function points have been tested.
- Two common forms of code coverage are:
- Function coverage, which reports on functions executed
- Statement coverage, which reports on the number of lines executed to complete the test
They both return a code coverage
Code coverage
Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing....
metric
Software metric
A software metric is a measure of some property of a piece of software or its specifications. Since quantitative measurements are essential in all sciences, there is a continuous effort by computer science practitioners and theoreticians to bring similar approaches to software development...
, measured as a percentage
Percentage
In mathematics, a percentage is a way of expressing a number as a fraction of 100 . It is often denoted using the percent sign, “%”, or the abbreviation “pct”. For example, 45% is equal to 45/100, or 0.45.Percentages are used to express how large/small one quantity is, relative to another quantity...
.
Black box testing
Black box testing treats the software as a "black box"—without any knowledge of internal implementation. Black box testing methods include: equivalence partitioningEquivalence partitioning
Equivalence partitioning is a software testing technique that divides the input data of a software unit into partitions of data from which test cases can be derived. In principle, test cases are designed to cover each partition at least once...
, boundary value analysis
Boundary value analysis
Boundary value analysis is a software testing technique in which tests are designed to include representatives of boundary values. Values on the minimum and maximum edges of an equivalence partition are tested. The values could be either input or output ranges of a software component...
, all-pairs testing
All-pairs testing
All-pairs testing or pairwise testing is a combinatorial software testing method that, for each pair of input parameters to a system , tests all possible discrete combinations of those parameters...
, fuzz testing
Fuzz testing
Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes or failing built-in code assertions...
, model-based testing
Model-based testing
Model-based testing is the application of Model based design for designing and optionally executing the necessary artifacts to perform software testing. Models can be used to represent the desired behavior of the System Under Test , or to represent the desired testing strategies and testing...
, exploratory testing and specification-based testing.
- Specification-based testing: Specification-based testing aims to test the functionality of software according to the applicable requirements. Thus, the tester inputs data into, and only sees the output from, the test object. This level of testing usually requires thorough test cases to be provided to the tester, who then can simply verify that for a given input, the output value (or behavior), either "is" or "is not" the same as the expected value specified in the test case.
- Specification-based testing is necessary, but it is insufficient to guard against certain risks.
- Advantages and disadvantages: The black box tester has no "bonds" with the code, and a tester's perception is very simple: a code must have bugs. Using the principle, "Ask and you shall receive," black box testers find bugs where programmers do not. On the other hand, black box testing has been said to be "like a walk in a dark labyrinth without a flashlight," because the tester doesn't know how the software being tested was actually constructed. As a result, there are situations when (1) a tester writes many test cases to check something that could have been tested by only one test case, and/or (2) some parts of the back-end are not tested at all.
Therefore, black box testing has the advantage of "an unaffiliated opinion", on the one hand, and the disadvantage of "blind exploring", on the other.
Grey box testing
Grey Box Testing (American spelling: gray box testing) involves having knowledge of internal data structures and algorithms for purposes of designing the test cases, but testing at the user, or black-box level. The tester is not required to have a full access to the software's source code. Manipulating input data and formatting output do not qualify as grey box, because the input and output are clearly outside of the "black-box" that we are calling the system under test. This distinction is particularly important when conducting integration testingIntegration testing
Integration testing is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before validation testing...
between two modules of code written by two different developers, where only the interfaces are exposed for test. However, modifying a data repository does qualify as grey box, as the user would not normally be able to change the data outside of the system under test. Grey box testing may also include reverse engineering to determine, for instance, boundary values or error messages.
By knowing the underlying concepts of how the software works, the tester makes better-informed testing choices while testing the software from outside. Typically, a grey box tester will be permitted to set up his testing environment; for instance, seeding a database
Database
A database is an organized collection of data for one or more purposes, usually in digital form. The data are typically organized to model relevant aspects of reality , in a way that supports processes requiring this information...
; and the tester can observe the state of the product being tested after performing certain actions. For instance, he/she may fire an SQL
SQL
SQL is a programming language designed for managing data in relational database management systems ....
query on the database and then observe the database, to ensure that the expected changes have been reflected. Grey box testings implements intelligent test scenarios, based on limited information. This will particularly apply to data type handling, exception handling, and so on.
Visual testing
The aim of visual testing is to provide developers with the ability to examine what was happening at the point of software failure by presenting the data in such a way that the developer can easily find the information he requires, and the information is expressed clearly.At the core of visual testing is the idea that showing someone a problem (or a test failure), rather than just describing it, greatly increases clarity and understanding. Visual testing therefore requires the recording of the entire test process – capturing everything that occurs on the test system in video format. Output videos are supplemented by real-time tester input via picture-in-a-picture webcam and audio commentary from microphones.
Visual testing provides a number of advantages. The quality of communication is increased dramatically because testers can show the problem (and the events leading up to it) to the developer as opposed to just describing it and the need to replicate test failures will cease to exist in many cases. The developer will have all the evidence he requires of a test failure and can instead focus on the cause of the fault and how it should be fixed.
Visual testing is particularly well-suited for environments that deploy agile methods in their development of software, since agile methods require greater communication between testers and developers and collaboration within small teams.
Ad hoc testing
Ad hoc testing
Ad hoc testing is a commonly used term for software testing performed without planning and documentation ....
and exploratory testing are important methodologies for checking software integrity, because they require less preparation time to implement, whilst important bugs can be found quickly. In ad hoc testing, where testing takes place in an improvised, impromptu way, the ability of a test tool to visually record everything that occurs on a system becomes very important.
Visual testing is gathering recognition in customer acceptance and usability testing
Usability testing
Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system...
, because the test can be used by many individuals involved in the development process. For the customer, it becomes easy to provide detailed bug reports and feedback, and for program users, visual testing can record user actions on screen, as well as their voice and image, to provide a complete picture at the time of software failure for the developer.
Testing levels
Tests are frequently grouped by where they are added in the software development process, or by the level of specificity of the test.The main levels during the development process as defined by the SWEBOK guide are unit-, integration-, and system testing that are distinguished by the test target without implying a specific process model. Other test levels are classified by the testing objective.
Unit testing
Unit testing, also known as component testing, refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors.These types of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. One function might have multiple tests, to catch corner case
Corner case
A corner case is a problem or situation that occurs only outside of normal operating parameters – specifically one that manifests itself when multiple environmental variables or conditions are simultaneously at extreme levels, even though each parameter is within the specified range for that...
s or other branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to assure that the building blocks the software uses work independently of each other.
Integration testing
Integration testing is any type of software testing that seeks to verify the interfaces between components against a software design. Software components may be integrated in an iterative way or all together ("big bang"). Normally the former is considered a better practice since it allows interface issues to be localised more quickly and fixed.Integration testing
Integration testing
Integration testing is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before validation testing...
works to expose defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a system.
System testing
System testingSystem testing
System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or...
tests a completely integrated system to verify that it meets its requirements.
System integration testing
System integration testingSystem integration testing
In the context of software systems and software engineering, system integration testing is a testing process that exercises a software system's coexistence with others. With multiple integrated systems, assuming that each have already passed system testing, SIT proceeds to test their required...
verifies that a system is integrated to any external or third-party systems defined in the system requirements.
Regression testing
Regression testing focuses on finding defects after a major code change has occurred. Specifically, it seeks to uncover software regressionSoftware regression
A software regression is a software bug which makes a feature stop functioning as intended after a certain event...
s, or old bugs that have come back. Such regressions occur whenever software functionality that was previously working correctly stops working as intended. Typically, regressions occur as an unintended consequence
Unintended consequence
In the social sciences, unintended consequences are outcomes that are not the outcomes intended by a purposeful action. The concept has long existed but was named and popularised in the 20th century by American sociologist Robert K. Merton...
of program changes, when the newly developed part of the software collides with the previously existing code. Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged. The depth of testing depends on the phase in the release process and the risk
Risk management
Risk management is the identification, assessment, and prioritization of risks followed by coordinated and economical application of resources to minimize, monitor, and control the probability and/or impact of unfortunate events or to maximize the realization of opportunities...
of the added features. They can either be complete, for changes added late in the release or deemed to be risky, to very shallow, consisting of positive tests on each feature, if the changes are early in the release or deemed to be of low risk.
Acceptance testing
Acceptance testing can mean one of two things:- A smoke test is used as an acceptance test prior to introducing a new build to the main testing process, i.e. before integrationIntegration testingIntegration testing is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before validation testing...
or regressionRegression testingRegression testing is any type of software testing that seeks to uncover new errors, or regressions, in existing functionality after changes have been made to a system, such as functional enhancements, patches or configuration changes....
. - Acceptance testing performed by the customer, often in their lab environment on their own hardware, is known as user acceptance testing (UAT). Acceptance testing may be performed as part of the hand-off process between any two phases of development.
Alpha testing
Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.Beta testing
Beta testing comes after alpha testing and can be considered a form of external user acceptance testing. Versions of the software, known as beta versions, are released to a limited audience outside of the programming team. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.Non-functional testing
Special methods exist to test non-functional aspects of software. In contrast to functional testing, which establishes the correct operation of the software (for example that it matches the expected behavior defined in the design requirements), non-functional testing verifies that the software functions properly even when it receives invalid or unexpected inputs. Software fault injectionFault injection
In software testing, fault injection is a technique for improving the coverage of a test by introducing faults to test code paths, in particular error handling code paths, that might otherwise rarely be followed. It is often used with stress testing and is widely considered to be an important part...
, in the form of fuzzing
Fuzz testing
Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes or failing built-in code assertions...
, is an example of non-functional testing. Non-functional testing, especially for software, is designed to establish whether the device under test can tolerate invalid or unexpected inputs, thereby establishing the robustness of input validation routines as well as error-management routines. Various commercial non-functional testing tools are linked from the software fault injection
Fault injection
In software testing, fault injection is a technique for improving the coverage of a test by introducing faults to test code paths, in particular error handling code paths, that might otherwise rarely be followed. It is often used with stress testing and is widely considered to be an important part...
page; there are also numerous open-source and free software tools available that perform non-functional testing.
Software performance testing
Performance testing is in general executed to determine how a system or sub-system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.Load testing is primarily concerned with testing that can continue to operate under a specific load, whether that be large quantities of data or a large number of users
Load testing
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any...
. This is generally referred to as software scalability
Scalability
In electronics scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth...
. The related load testing activity of when performed as a non-functional activity is often referred to as endurance testing. Volume testing
Volume testing
Volume Testing belongs to the group of non-functional tests, which are often misunderstood and/or used interchangeably. Volume testing refers to testing a software application with a certain amount of data. This amount can, in generic terms, be the database size or it could also be the size of an...
is a way to test functionality. Stress testing
Stress testing
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results...
is a way to test reliability under unexpected or rare workloads. Stability testing (often referred to as load or endurance testing) checks to see if the software can continuously function well in or above an acceptable period.
There is little agreement on what the specific goals of performance testing are. The terms load testing, performance testing, reliability testing, and volume testing, are often used interchangeably.
Usability testing
Usability testingUsability testing
Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system...
is needed to check if the user interface is easy to use and understand. It is concerned mainly with the use of the application.
Security testing
Security testingSecurity Testing
Security testing is a process to determine that an information system protects data and maintains functionality as intended.The six basic security concepts that need to be covered by security testing are: confidentiality, integrity, authentication, availability, authorization and non-repudiation...
is essential for software that processes confidential data to prevent system intrusion by hackers
Hacker (computer security)
In computer security and everyday language, a hacker is someone who breaks into computers and computer networks. Hackers may be motivated by a multitude of reasons, including profit, protest, or because of the challenge...
.
Internationalization and localization
The general ability of software to be internationalized and localizedInternationalization and localization
In computing, internationalization and localization are means of adapting computer software to different languages, regional differences and technical requirements of a target market...
can be automatically tested without actual translation, by using pseudolocalization
Pseudolocalization
Pseudolocalization is a software testing method that is used to test internationalization aspects of software. Specifically, it brings to light potential difficulties with localization by replacing localizable text with text that imitates the most problematic characteristics of text from a wide...
. It will verify that the application still works, even after it has been translated into a new language or adapted for a new culture (such as different currencies or time zones).
Actual translation to human languages must be tested, too. Possible localization failures include:
- Software is often localized by translating a list of strings out of context, and the translator may choose the wrong translation for an ambiguous source string.
- Technical terminology may become inconsistent if the project is translated by several people without proper coordination or if the translator is imprudent.
- Literal word-for-word translations may sound inappropriate, artificial or too technical in the target language.
- Untranslated messages in the original language may be left hard coded in the source code.
- Some messages may be created automatically at run time and the resulting string may be ungrammatical, functionally incorrect, misleading or confusing.
- Software may use a keyboard shortcutKeyboard shortcutIn computing, a keyboard shortcut is a finite set of one or more keys that invoke a software or operating system operation when triggered by the user. A meaning of term "keyboard shortcut" can vary depending on software manufacturer...
which has no function on the source language's keyboard layoutKeyboard layoutA keyboard layout is any specific mechanical, visual, or functional arrangement of the keys, legends, or key–meaning associations of a computer, typewriter, or other typographic keyboard....
, but is used for typing characters in the layout of the target language. - Software may lack support for the character encodingCharacter encodingA character encoding system consists of a code that pairs each character from a given repertoire with something else, such as a sequence of natural numbers, octets or electrical pulses, in order to facilitate the transmission of data through telecommunication networks or storage of text in...
of the target language. - Fonts and font sizes which are appropriate in the source language, may be inappropriate in the target language; for example, CJK characters may become unreadable if the font is too small.
- A string in the target language may be longer than the software can handle. This may make the string partly invisible to the user or cause the software to crash or malfunction.
- Software may lack proper support for reading or writing bi-directional textBi-directional textBi-directional text is text containing text in both text directionalities, both right-to-left and left-to-right . It generally involves text containing different types of alphabets, but may also refer to boustrophedon, which is changing text directionality in each row.Some writing systems of the...
. - Software may display images with text that wasn't localized.
- Localized operating systems may have differently-named system configuration fileConfiguration fileIn computing, configuration files, or config files configure the initial settings for some computer programs. They are used for user applications, server processes and operating system settings. The files are often written in ASCII and line-oriented, with lines terminated by a newline or carriage...
s and environment variableEnvironment variableEnvironment variables are a set of dynamic named values that can affect the way running processes will behave on a computer.They can be said in some sense to create the operating environment in which a process runs...
s and different formats for dateDate and time notation by countryDifferent conventions exist around the world for date and time representation, both written and spoken.- Differences :Differences can exist in:*The calendar that is used.*The order in which the year, month and day are represented....
and currencyCurrencyIn economics, currency refers to a generally accepted medium of exchange. These are usually the coins and banknotes of a particular government, which comprise the physical aspects of a nation's money supply...
.
To avoid these and other localization problems, a tester who knows the target language must run the program with all the possible use cases for translation to see if the messages are readable, translated correctly in context and don't cause failures.
Destructive testing
Destructive testing attempts to cause the software or a sub-system to fail, in order to test its robustness.Traditional CMMI or waterfall development model
A common practice of software testing is that testing is performed by an independent group of testers after the functionality is developed, before it is shipped to the customer. This practice often results in the testing phase being used as a projectProject management
Project management is the discipline of planning, organizing, securing, and managing resources to achieve specific goals. A project is a temporary endeavor with a defined beginning and end , undertaken to meet unique goals and objectives, typically to bring about beneficial change or added value...
buffer to compensate for project delays, thereby compromising the time devoted to testing.
Another practice is to start software testing at the same moment the project starts and it is a continuous process until the project finishes.
Agile or Extreme development model
In contrast, some emerging software disciplines such as extreme programmingExtreme Programming
Extreme programming is a software development methodology which is intended to improve software quality and responsiveness to changing customer requirements...
and the agile software development
Agile software development
Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams...
movement, adhere to a "test-driven software development
Test-driven development
Test-driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new...
" model. In this process, unit tests are written first, by the software engineers
Software engineering
Software Engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software...
(often with pair programming
Pair programming
Pair programming is an agile software development technique in which two programmers work together at one workstation. One, the driver, types in code while the other, the observer , reviews each line of code as it is typed in...
in the extreme programming methodology). Of course these tests fail initially; as they are expected to. Then as code is written it passes incrementally larger portions of the test suites. The test suites are continuously updated as new failure conditions and corner cases are discovered, and they are integrated with any regression tests that are developed. Unit tests are maintained along with the rest of the software source code and generally integrated into the build process (with inherently interactive tests being relegated to a partially manual build acceptance process). The ultimate goal of this test process is to achieve continuous deployment where software updates can be published to the public frequently.
A sample testing cycle
Although variations exist between organizations, there is a typical cycle for testing. The sample below is common among organizations employing the Waterfall development model.- Requirements analysisRequirements analysisRequirements analysis in systems engineering and software engineering, encompasses those tasks that go into determining the needs or conditions to meet for a new or altered product, taking account of the possibly conflicting requirements of the various stakeholders, such as beneficiaries or users...
: Testing should begin in the requirements phase of the software development life cycle. During the design phase, testers work with developers in determining what aspects of a design are testable and with what parameters those tests work. - Test planning: Test strategyTest strategyA test strategy is an outline that describes the testing portion of the software development cycle. It is created to inform project managers, testers, and developers about some key issues of the testing process...
, test planTest planA test plan is a document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be.-Test plans:...
, testbedTestbedA testbed is a platform for experimentation of large development projects. Testbeds allow for rigorous, transparent, and replicable testing of scientific theories, computational tools, and new technologies.The term is used across many disciplines to describe a development environment that is...
creation. Since many activities will be carried out during testing, a plan is needed. - Test development: Test procedures, test scenarios, test caseTest caseA test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
s, test datasets, test scripts to use in testing software. - Test execution: Testers execute the software based on the plans and test documents then report any errors found to the development team.
- Test reporting: Once testing is completed, testers generate metrics and make final reports on their test effortTest effortIn software development, test effort refers to the expenses for tests. There is a relation with test costs and failure costs...
and whether or not the software tested is ready for release. - Test result analysis: Or Defect Analysis, is done by the development team usually along with the client, in order to decide what defects should be assigned, fixed, rejected (i.e. found software working properly) or deferred to be dealt with later.
- Defect Retesting: Once a defect has been dealt with by the development team, it is retested by the testing team. AKA Resolution testing.
- Regression testing: It is common to have a small test program built of a subset of tests, for each integration of new, modified, or fixed software, in order to ensure that the latest delivery has not ruined anything, and that the software product as a whole is still working correctly.
- Test Closure: Once the test meets the exit criteria, the activities such as capturing the key outputs, lessons learned, results, logs, documents related to the project are archived and used as a reference for future projects.
Automated testing
Many programming groups are relying more and more on automated testingTest automation
Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions...
, especially groups that use
test-driven development
Test-driven development
Test-driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new...
. There are many frameworks to write tests in, and continuous integration
Continuous integration
In software engineering, continuous integration implements continuous processes of applying quality control — small pieces of effort, applied frequently...
software will run tests automatically every time code is checked into a version control system.
While automation cannot reproduce everything that a human can do (and all the ways they think of doing it), it can be very useful for regression testing. However, it does require a well-developed test suite
Test suite
In software development, a test suite, less commonly known as a validation suite, is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviours. A test suite often contains detailed instructions or goals for each...
of testing scripts in order to be truly useful.
Testing tools
Program testing and fault detection can be aided significantly by testing tools and debuggerDebugger
A debugger or debugging tool is a computer program that is used to test and debug other programs . The code to be examined might alternatively be running on an instruction set simulator , a technique that allows great power in its ability to halt when specific conditions are encountered but which...
s.
Testing/debug tools include features such as:
- Program monitors, permitting full or partial monitoring of program code including:
- Instruction set simulatorInstruction Set SimulatorAn instruction set simulator is a simulation model, usually coded in a high-level programming language, which mimics the behavior of a mainframe or microprocessor by "reading" instructions and maintaining internal variables which represent the processor's registers.Instruction simulation is a...
, permitting complete instruction level monitoring and trace facilities - Program animationProgram animationProgram animation or Stepping refers to the very common debugging method of executing code one "line" at a time. The programmer may examine the state of the program, machine, and related data before and after execution of a particular line of code...
, permitting step-by-step execution and conditional breakpointBreakpointIn software development, a breakpoint is an intentional stopping or pausing place in a program, put in place for debugging purposes. It is also sometimes simply referred to as a pause....
at source level or in machine codeMachine codeMachine code or machine language is a system of impartible instructions executed directly by a computer's central processing unit. Each instruction performs a very specific task, typically either an operation on a unit of data Machine code or machine language is a system of impartible instructions... - Code coverageCode coverageCode coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing....
reports
- Instruction set simulator
- Formatted dump or symbolic debugging, tools allowing inspection of program variables on error or at chosen points
- Automated functional GUI testing tools are used to repeat system-level tests through the GUI
- BenchmarkBenchmark (computing)In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it...
s, allowing run-time performance comparisons to be made - Performance analysisPerformance analysisIn software engineering, profiling is a form of dynamic program analysis that measures, for example, the usage of memory, the usage of particular instructions, or frequency and duration of function calls...
(or profiling tools) that can help to highlight hot spotHot spot (computer science)A hot spot in computer science is most usually defined as a region of a computer program where a high proportion of executed instructions occur or where most time is spent during the program's execution .If a program is stopped randomly, the program counter...
s and resource usage
Some of these features may be incorporated into an Integrated Development Environment
Integrated development environment
An integrated development environment is a software application that provides comprehensive facilities to computer programmers for software development...
(IDE).
- A regression testing technique is to have a standard set of tests, which cover existing functionality that result in persistent tabular data, and to compare pre-change data to post-change data, where there should not be differences, using a tool like diffkit. Differences detected indicate unexpected functionality changes or "regression".
Measurement in software testing
Usually, quality is constrained to such topics as correctness, completeness, securityComputer security audit
A computer security audit is a manual or systematic measurable technical assessment of a system or application. Manual assessments include interviewing staff, performing security vulnerability scans, reviewing application and operating system access controls, and analyzing physical access to the...
, but can also include more technical requirements as described under the ISO
International Organization for Standardization
The International Organization for Standardization , widely known as ISO, is an international standard-setting body composed of representatives from various national standards organizations. Founded on February 23, 1947, the organization promulgates worldwide proprietary, industrial and commercial...
standard ISO/IEC 9126, such as capability, reliability
Reliability engineering
Reliability engineering is an engineering field, that deals with the study, evaluation, and life-cycle management of reliability: the ability of a system or component to perform its required functions under stated conditions for a specified period of time. It is often measured as a probability of...
, efficiency
Algorithmic efficiency
In computer science, efficiency is used to describe properties of an algorithm relating to how much of various types of resources it consumes. Algorithmic efficiency can be thought of as analogous to engineering productivity for a repeating or continuous process, where the goal is to reduce...
, portability
Porting
In computer science, porting is the process of adapting software so that an executable program can be created for a computing environment that is different from the one for which it was originally designed...
, maintainability
Maintainability
In engineering, maintainability is the ease with which a product can be maintained in order to:* isolate defects or their cause* correct defects or their cause* meet new requirements* make future maintenance easier, or* cope with a changed environment...
, compatibility, and usability
Usability
Usability is the ease of use and learnability of a human-made object. The object of use can be a software application, website, book, tool, machine, process, or anything a human interacts with. A usability study may be conducted as a primary job function by a usability analyst or as a secondary job...
.
There are a number of frequently-used software measures, often called metrics, which are used to assist in determining the state of the software or the adequacy of the testing.
Testing artifacts
The software testing process can produce several artifactsArtifact (software development)
An artifact is one of many kinds of tangible by-product produced during the development of software. Some artifacts help describe the function, architecture, and design of software...
.
Test plan: A test specification is called a test plan
Test plan
A test plan is a document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be.-Test plans:...
. The developers are well aware what test plans will be executed and this information is made available to management and the developers. The idea is to make them more cautious when developing their code or making additional changes. Some companies have a higher-level document called a test strategy
Test strategy
A test strategy is an outline that describes the testing portion of the software development cycle. It is created to inform project managers, testers, and developers about some key issues of the testing process...
.
Traceability matrix: A traceability matrix
Traceability matrix
A traceability matrix is a document, usually in the form of a table, that correlates any two baselined documents that require a many to many relationship to determine the completeness of the relationship...
is a table that correlates requirements or design documents to test documents. It is used to change tests when related source documents are changed, to select test cases for execution when planning for regression tests by considering requirement coverage.
Test case: A test case
Test case
A test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
normally consists of a unique identifier, requirement references from a design specification, preconditions, events, a series of steps (also known as actions) to follow, input, output, expected result, and actual result. Clinically defined a test case is an input and an expected result. This can be as pragmatic as 'for condition x your derived result is y', whereas other test cases described in more detail the input scenario and what results might be expected. It can occasionally be a series of steps (but often steps are contained in a separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one expected result or expected outcome. The optional fields are a test case ID, test step, or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database, or other common repository. In a database system, you may also be able to see past test results, who generated the results, and what system configuration was used to generate those results. These past results would usually be stored in a separate table.
Test script: A test script
Test script
A test script in software testing is a set of instructions that will be performed on the system under test to test that the system functions as expected.There are various means for executing test scripts.* Manual testing...
is a procedure, or programing code that replicates user actions. Initially the term was derived from the product of work created by automated regression test tools. Test Case will be a baseline to create test scripts using a tool or a program.
Test suite: The most common term for a collection of test cases is a test suite
Test suite
In software development, a test suite, less commonly known as a validation suite, is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviours. A test suite often contains detailed instructions or goals for each...
. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.
Test data: In most cases, multiple sets of values or data are used to test the same functionality of a particular feature. All the test values and changeable environmental components are collected in separate files and stored as test data. It is also useful to provide this data to the client and with the product or a project.
Test harness: The software, tools, samples of data input and output, and configurations are all referred to collectively as a test harness
Test harness
In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the Test execution engine and the Test script...
.
Certifications
Several certification programs exist to support the professional aspirations of software testers and quality assurance specialists. No certification currently offered actually requires the applicant to demonstrate the ability to test software. No certification is based on a widely accepted body of knowledge. This has led some to declare that the testing field is not ready for certification. Certification itself cannot measure an individual's productivity, their skill, or practical knowledge, and cannot guarantee their competence, or professionalism as a tester.Software testing certification types
- Exam-based: Formalized exams, which need to be passed; can also be learned by self-study [e.g., for ISTQBISTQBThe International Software Testing Qualifications Board is a software testing qualification certification organisation. Founded in Edinburgh in November 2002, ISTQB is a non-profit association legally registered in Belgium....
or QAIQAIQAI is a global management consulting and workforce development organization. Set up in 1980 in Orlando, Florida, it caters to Operational Excellence across organizations worldwide.-QAI Global Services:...
] - Education-based: Instructor-led sessions, where each course has to be passed [e.g., International Institute for Software Testing (IIST)].
Testing certifications
- Certified Associate in Software Testing (CAST) offered by the QAIQAIQAI is a global management consulting and workforce development organization. Set up in 1980 in Orlando, Florida, it caters to Operational Excellence across organizations worldwide.-QAI Global Services:...
- CATe offered by the International Institute for Software Testing
- Certified Manager in Software Testing (CMST) offered by the QAIQAIQAI is a global management consulting and workforce development organization. Set up in 1980 in Orlando, Florida, it caters to Operational Excellence across organizations worldwide.-QAI Global Services:...
- Certified Software Tester (CSTE) offered by the Quality Assurance Institute (QAI)
- Certified Software Test Professional (CSTP) offered by the International Institute for Software Testing
- CSTP (TM) (Australian Version) offered by K. J. Ross & Associates
- ISEB offered by the Information Systems Examinations BoardISEBThe Information Systems Examinations Board is an examination awarding body and a part of BCS, The Chartered Institute for IT.Initially started as a collaboration between the National Computing Centre and BCS for the creation of the 'Certificate in Systems Analysis and Design' for the then Systems...
- ISTQB Certified Tester, Foundation Level (CTFL) offered by the International Software Testing Qualification BoardISTQBThe International Software Testing Qualifications Board is a software testing qualification certification organisation. Founded in Edinburgh in November 2002, ISTQB is a non-profit association legally registered in Belgium....
- ISTQB Certified Tester, Advanced Level (CTAL) offered by the International Software Testing Qualification BoardISTQBThe International Software Testing Qualifications Board is a software testing qualification certification organisation. Founded in Edinburgh in November 2002, ISTQB is a non-profit association legally registered in Belgium....
- TMPF TMap Next Foundation offered by the Examination Institute for Information Science
- TMPA TMap Next Advanced offered by the Examination Institute for Information Science
Quality assurance certifications
- CMSQ offered by the Quality Assurance Institute (QAI).
- CSQA offered by the Quality Assurance Institute (QAI)
- CSQE offered by the American Society for QualityAmerican Society for QualityAmerican Society for Quality , formerly known as American Society for Quality Control , is a knowledge-based global community of quality control experts, with nearly 85,000 members dedicated to the promotion and advancement of quality tools, principles, and practices in their workplaces and in...
(ASQ) - CQIA offered by the American Society for QualityAmerican Society for QualityAmerican Society for Quality , formerly known as American Society for Quality Control , is a knowledge-based global community of quality control experts, with nearly 85,000 members dedicated to the promotion and advancement of quality tools, principles, and practices in their workplaces and in...
(ASQ)
Controversy
Some of the major software testing controversiesSoftware testing controversies
There is considerable variety among software testing writers and consultants about what constitutes responsible software testing. Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester...
include:
What constitutes responsible software testing? : Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester to select or invent testing practices to suit each unique situation.
Agile vs. traditional : Should testers learn to work under conditions of uncertainty and constant change or should they aim at process "maturity"
Capability Maturity Model
The Capability Maturity Model is a development model that was created after study of data collected from organizations that contracted with the U.S. Department of Defense, who funded the research. This model became the foundation from which CMU created the Software Engineering Institute...
? The agile testing
Agile testing
Agile testing is a software testing practice that follows the principles of agile software development. Agile testing does not emphasize testing procedures and focuses on ongoing testing against newly developed code until quality software from an end customer's perspective results...
movement has received growing popularity since 2006 mainly in commercial circles, whereas government and military software providers use this methodology but also the traditional test-last models (e.g. in the Waterfall model
Waterfall model
The waterfall model is a sequential design process, often used in software development processes, in which progress is seen as flowing steadily downwards through the phases of Conception, Initiation, Analysis, Design, Construction, Testing, Production/Implementation and Maintenance.The waterfall...
).
Exploratory test vs. scripted : Should tests be designed at the same time as they are executed or should they be designed beforehand?
Manual testing vs. automated : Some writers believe that test automation
Test automation
Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions...
is so expensive relative to its value that it should be used sparingly. More in particular, test-driven development
Test-driven development
Test-driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new...
states that developers should write unit-tests of the XUnit
XUnit
Various code-driven testing frameworks have come to be known collectively as xUnit. These frameworks allow testing of different elements of software, such as functions and classes...
type before coding the functionality. The tests then can be considered as a way to capture and implement the requirements.
Software design vs. software implementation : Should testing be carried out only at the end or throughout the whole process?
Who watches the watchmen? : The idea is that any form of observation is also an interaction—the act of testing can also affect that which is being tested.
See also
- Acceptance testing
- All-pairs testingAll-pairs testingAll-pairs testing or pairwise testing is a combinatorial software testing method that, for each pair of input parameters to a system , tests all possible discrete combinations of those parameters...
- Automated testing
- Dynamic program analysisDynamic program analysisDynamic program analysis is the analysis of computer software that is performed by executing programs built from that software system on a real or virtual processor. For dynamic program analysis to be effective, the target program must be executed with sufficient test inputs to produce interesting...
- Formal verificationFormal verificationIn the context of hardware and software systems, formal verification is the act of proving or disproving the correctness of intended algorithms underlying a system with respect to a certain formal specification or property, using formal methods of mathematics .- Usage :Formal verification can be...
- GUI software testingGUI software testingIn software engineering, graphical user interface testing is the process of testing a product's graphical user interface to ensure it meets its written specifications...
- Independent test organizationIndependent test organizationAn independent test organization is an organization, person, or company that tests products, materials, software, etc. according to agreed requirements. The test organization can be affiliated with the government or universities or can be an independent testing laboratory...
- Manual testingManual testingManual testing is the process of manually testing software for defects. It requires a tester to play the role of an end user, and use most of all features of the application to ensure correct behavior...
- Orthogonal array testing
- Pair TestingPair TestingPair testing is a software development technique in which two team members work together at one keyboard to test the software application. One does the testing and the other analyzes or reviews the testing...
- Reverse semantic traceabilityReverse semantic traceabilityReverse Semantic Traceability is a quality control method for verification improvement that helps to insure high quality of artifacts by backward translation at each stage of the software development process.- Brief introduction :...
- Software testabilitySoftware testabilitySoftware testability is the degree to which a software artifact supports testing in a given test context....
- Static code analysisStatic code analysisStatic program analysis is the analysis of computer software that is performed without actually executing programs built from that software In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code...
- Test management toolsTest management toolsTest management tools are used to structure automated tests and manual test processes, and to easily manage multiple environments. Quality assurance teams use these types of tools as a single application for managing test cases, environments, automated tests, defects and project tasks...
- Web testingWeb testingWeb testing is the name given to software testing that focuses on web applications. Complete testing of a web-based system before going live can help address issues before the system is revealed to the public...