Performance testing
Encyclopedia
In software engineering
, performance testing is in general testing
performed to determine how a system
performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attribute
s of the system, such as scalability
, reliability
and resource usage.
Performance testing is a subset of Performance engineering
, an emerging computer science
practice which strives to build performance into the design and architecture of a system, prior to the onset of actual coding effort.
is the simplest form of performance testing. A load test is usually conducted to understand the behavior of the application under a specific expected load. This load can be the expected concurrent number of users on the application
performing a specific number of transaction
s within the set duration. This test will give out the response times of all the important business critical transactions. If the database
, application server
, etc. are also monitored, then this simple test can itself point towards any bottleneck
s in the application software....
is normally used to understand the upper limits of capacity within the application landscape. This kind of test is done to determine the application's robustness in terms of extreme load and helps application administrators to determine if the application will perform sufficiently if the current load goes well above the expected maximum.
Many performance tests are undertaken without due consideration to the setting of realistic performance goals. The first question from a business perspective should always be "why are we performance testing?". These considerations are part of the business case
of the testing.
Performance goals will differ depending on the application technology and purpose however they should always include some of the following:
especially if the iterative part contains the Login & Logout activity
If your application has no concept of end-users then your performance goal is likely to be based on a maximum throughput or transaction rate. A common example would be casual browsing of a web site such as Wikipedia.
In terms of response time this is what all load testing
tools actually measure.
It may be relevant to set server response time goals between all nodes of the application landscape.
apart from recognizing a period of time where there is no activity 'on the wire'.
To measure render response time it is generally necessary to include functional test script
s as part of the performance test scenario which is a feature not offered by many load testing tools.
for more details.
However, performance testing is frequently not performed against a specification i.e. no one will have expressed what the maximum acceptable response time for a given population of users should be. Performance testing is frequently used as part of the process of performance profile tuning. The idea is to identify the “weakest link” – there is inevitably a part of the system which, if it is made to respond faster, will result in the overall system running faster. It is sometimes a difficult task to identify which part of the system represents this critical path, and some test tools include (or can have add-ons that provide) instrumentation that runs on the server (agents) and report transaction times, database access times, network overhead, and other server monitors, which can be analyzed together with the raw performance statistics. Without such instrumentation one might have to have someone crouched over Windows Task Manager
at the server to see how much CPU load the performance tests are generating (assuming a Windows system is under test).
Performance testing can be performed across the web, and even done in different parts of the country, since it is known that the response times of the internet itself vary regionally. It can also be done in-house, although routers would then need to be configured to introduce the lag what would typically occur on public networks. Loads should be introduced to the system from realistic points. For example, if 50% of a system's user base will be accessing the system via a 56K modem connection and the other half over a T1
, then the load injectors (computers that simulate real users) should either inject load over the same connections (ideal) or simulate the network latency of such connections, following the same user profile.
It is always helpful to have a statement of the likely peak numbers of users that might be expected to use the system at peak times. If there can also be a statement of what constitutes the maximum allowable 95 percentile response time, then an injector configuration could be used to test whether the proposed system met that specification.
The performance testing environment should not be clubbed with User acceptance testing (UAT) or development environment. This is dangerous as if an UAT or Integration test or other tests are going on in the same environment, then the results obtained from the performance testing may not be reliable. As a best practice it is always advisable to have a separate performance testing environment resembling the production environment as much as possible.
Loosely-coupled architectural implementations (e.g.: SOA
) have created additional complexities with performance testing. Enterprise services or assets (that share a common infrastructure
or platform) require coordinated performance testing (with all consumers creating production-like transaction volumes and load on shared infrastructures or platforms) to truly replicate production-like states. Due to the complexity and financial and time requirements around this activity, some organizations now employ tools that can monitor and create production-like conditions (also referred as "noise") in their performance testing environments (PTE
) to understand capacity and resource requirements and verify / validate quality attributes.
s use tools such as profilers to measure what parts of a device or software contributes most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time.
1. Performance Testing is done to break the system.
Stress Testing is done to understand the break point of the system. Otherwise normal load testing is generally done to understand the behavior of the application under the expected user load. Depending on other requirements, such as expectation of spike load, continued load for an extended period of time would demand spike, endurance soak or stress testing.
2. Performance Testing should only be done after the System Integration Testing
Although this is mostly the norm in the industry, performance testing can also be done while the initial development of the application is taking place. This kind of approach is known as the Early Performance Testing. This approach would ensure a holistic development of the application keeping the performance parameters in mind. Thus the finding of a performance bug just before the release of the application and the cost involved in rectifying the bug is reduced to a great extent.
3. Performance Testing only involves creation of scripts and any application changes would cause a simple refactoring of the scripts.
Performance Testing in itself is an evolving science in the Software Industry
. Scripting itself although important, is only one of the components of the performance testing. The major challenge for any performance tester is to determine the type of tests needed to execute and analyzing the various performance counters to determine the performance bottleneck.
The other segment of the myth concerning the change in application would result only in little refactoring in the scripts is also untrue as any form of change on the UI especially in the Web protocol would entail complete re-development of the scripts from scratch. This problem becomes bigger if the protocols involved include Web Services, Siebel
, Citrix, and SAP
.
Performance testing can be combined with stress testing
, in order to see what happens when an acceptable load is exceeded –does the system crash? How long does it take to recover if a large load is reduced? Does it fail in a way that causes collateral damage?
Analytical Performance Modeling is a method to model the behaviour of an application in a spreadsheet. The model is fed with measurements of transaction resource demands (CPU, disk I/O, LAN
, WAN
), weighted by the transaction-mix (business transactions per hour). The weighted transaction resource demands are added-up to obtain the hourly resource demands and divided by the hourly resource capacity to obtain the resource loads. Using the responsetime formula (R=S/(1-U), R=responsetime, S=servicetime, U=load), responsetimes can be calculated and calibrated with the results of the performance tests. Analytical performance modelling allows evaluation of design options and system sizing based on actual or anticipated business usage. It is therefore much faster and cheaper than performance testing, though it requires thorough understanding of the hardware platforms.
Software engineering
Software Engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software...
, performance testing is in general testing
Software testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
performed to determine how a system
System
System is a set of interacting or interdependent components forming an integrated whole....
performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attribute
Attribute
Attribute may refer to:* In research, a characteristic of an object - see attribute * In philosophy, property , an abstraction of a characteristic of an entity or substance...
s of the system, such as scalability
Scalability
In electronics scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth...
, reliability
Reliability
In general, reliability is the ability of a person or system to perform and maintain its functions in routine circumstances, as well as hostile or unexpected circumstances.Reliability may refer to:...
and resource usage.
Performance testing is a subset of Performance engineering
Performance Engineering
Performance engineering within systems engineering, encompasses the set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the Systems Development Life Cycle which ensures that a solution will be designed, implemented, and operationally supported to meet the...
, an emerging computer science
Computer science
Computer science or computing science is the study of the theoretical foundations of information and computation and of practical techniques for their implementation and application in computer systems...
practice which strives to build performance into the design and architecture of a system, prior to the onset of actual coding effort.
Load testing
Load testingLoad testing
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any...
is the simplest form of performance testing. A load test is usually conducted to understand the behavior of the application under a specific expected load. This load can be the expected concurrent number of users on the application
Application software
Application software, also known as an application or an "app", is computer software designed to help the user to perform specific tasks. Examples include enterprise software, accounting software, office suites, graphics software and media players. Many application programs deal principally with...
performing a specific number of transaction
Transaction processing
In computer science, transaction processing is information processing that is divided into individual, indivisible operations, called transactions. Each transaction must succeed or fail as a complete unit; it cannot remain in an intermediate state...
s within the set duration. This test will give out the response times of all the important business critical transactions. If the database
Database
A database is an organized collection of data for one or more purposes, usually in digital form. The data are typically organized to model relevant aspects of reality , in a way that supports processes requiring this information...
, application server
Application server
An application server is a software framework that provides an environment in which applications can run, no matter what the applications are or what they do...
, etc. are also monitored, then this simple test can itself point towards any bottleneck
Bottleneck
A bottleneck is a phenomenon where the performance or capacity of an entire system is limited by a single or limited number of components or resources. The term bottleneck is taken from the 'assets are water' metaphor. As water is poured out of a bottle, the rate of outflow is limited by the width...
s in the application software....
Stress testing
Stress testingStress testing
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results...
is normally used to understand the upper limits of capacity within the application landscape. This kind of test is done to determine the application's robustness in terms of extreme load and helps application administrators to determine if the application will perform sufficiently if the current load goes well above the expected maximum.
Endurance testing (soak testing)
Endurance testing is usually done to determine if the application can sustain the continuous expected load. During endurance tests, memory utilization is monitored to detect potential leaks. Also important, but often overlooked is performance degradation. That is, to ensure that the throughput and/or response times after some long period of sustained activity are as good or better than at the beginning of the test. It essentially involves applying a significant load to a system for an extended, significant period of time. The goal is to discover how the system behaves under sustained use.Spike testing
Spike testing, as the name suggests is done by spiking the number of users and understanding the behavior of the application; whether performance will suffer, the application will fail, or it will be able to handle dramatic changes in load.Configuration testing
Configuration testing is another variation on traditional performance testing. Rather than testing for performance from the perspective of load you are testing the effects of configuration changes in the application landscape on application performance and behaviour. A common example would be experimenting with different methods of load-balancing.Isolation testing
Isolation testing is not unique to performance testing but a term used to describe repeating a test execution that resulted in an application problem. Often used to isolate and confirm the fault domain.Setting performance goals
Performance testing can serve different purposes.- It can demonstrate that the system meets performance criteria.
- It can compare two systems to find which performs better.
- Or it can measure what parts of the system or workload causes the system to perform badly.
Many performance tests are undertaken without due consideration to the setting of realistic performance goals. The first question from a business perspective should always be "why are we performance testing?". These considerations are part of the business case
Business case
A business case captures the reasoning for initiating a project or task. It is often presented in a well-structured written document, but may also sometimes come in the form of a short verbal argument or presentation. The logic of the business case is that, whenever resources such as money or...
of the testing.
Performance goals will differ depending on the application technology and purpose however they should always include some of the following:
Concurrency/throughput
If an application identifies end-users by some form of login procedure then a concurrency goal is highly desirable. By definition this is the largest number of concurrent application users that the application is expected to support at any given moment. The work-flow of your scripted transaction may impact true application concurrencyConcurrency
Concurrency, concurrent, or concurrence may refer to:* Concurrence, a legal term referring to the need to prove both actus reus and mens rea...
especially if the iterative part contains the Login & Logout activity
If your application has no concept of end-users then your performance goal is likely to be based on a maximum throughput or transaction rate. A common example would be casual browsing of a web site such as Wikipedia.
Server response time
This refers to the time taken for one application node to respond to the request of another. A simple example would be a HTTP 'GET' request from browser client to web server.In terms of response time this is what all load testing
Load testing
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any...
tools actually measure.
It may be relevant to set server response time goals between all nodes of the application landscape.
Render response time
A difficult thing for load testing tools to deal with as they generally have no concept of what happens within a nodeNode (networking)
In communication networks, a node is a connection point, either a redistribution point or a communication endpoint . The definition of a node depends on the network and protocol layer referred to...
apart from recognizing a period of time where there is no activity 'on the wire'.
To measure render response time it is generally necessary to include functional test script
Test script
A test script in software testing is a set of instructions that will be performed on the system under test to test that the system functions as expected.There are various means for executing test scripts.* Manual testing...
s as part of the performance test scenario which is a feature not offered by many load testing tools.
Performance specifications
It is critical to detail performance specifications (requirements) and document them in any performance test plan. Ideally, this is done during the requirements development phase of any system development project, prior to any design effort. See Performance EngineeringPerformance Engineering
Performance engineering within systems engineering, encompasses the set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the Systems Development Life Cycle which ensures that a solution will be designed, implemented, and operationally supported to meet the...
for more details.
However, performance testing is frequently not performed against a specification i.e. no one will have expressed what the maximum acceptable response time for a given population of users should be. Performance testing is frequently used as part of the process of performance profile tuning. The idea is to identify the “weakest link” – there is inevitably a part of the system which, if it is made to respond faster, will result in the overall system running faster. It is sometimes a difficult task to identify which part of the system represents this critical path, and some test tools include (or can have add-ons that provide) instrumentation that runs on the server (agents) and report transaction times, database access times, network overhead, and other server monitors, which can be analyzed together with the raw performance statistics. Without such instrumentation one might have to have someone crouched over Windows Task Manager
Windows Task Manager
Windows Task Manager is a task manager application included with the Microsoft Windows NT family of operating systems that provides detailed information about computer performance and running applications, processes and CPU usage, commit charge and memory information, network activity and...
at the server to see how much CPU load the performance tests are generating (assuming a Windows system is under test).
Performance testing can be performed across the web, and even done in different parts of the country, since it is known that the response times of the internet itself vary regionally. It can also be done in-house, although routers would then need to be configured to introduce the lag what would typically occur on public networks. Loads should be introduced to the system from realistic points. For example, if 50% of a system's user base will be accessing the system via a 56K modem connection and the other half over a T1
T1
- Technology :* T1 font or Cork encoding, a character encoding used in computer programming and telecommunications* UltraSPARC T1, a microprocessor* T1, another name for Digital Signal 1 , a standard digital communications link...
, then the load injectors (computers that simulate real users) should either inject load over the same connections (ideal) or simulate the network latency of such connections, following the same user profile.
It is always helpful to have a statement of the likely peak numbers of users that might be expected to use the system at peak times. If there can also be a statement of what constitutes the maximum allowable 95 percentile response time, then an injector configuration could be used to test whether the proposed system met that specification.
Questions to ask
Performance specifications should ask the following questions, at a minimum:- In detail, what is the performance test scope? What subsystems, interfaces, components, etc. are in and out of scope for this test?
- For the user interfaces (UIs) involved, how many concurrent users are expected for each (specify peak vs. nominal)?
- What does the target system (hardware) look like (specify all server and network appliance configurations)?
- What is the Application Workload Mix of each application component? (for example: 20% login, 40% search, 30% item select, 10% checkout).
- What is the System Workload Mix? [Multiple workloads may be simulated in a single performance test] (for example: 30% Workload A, 20% Workload B, 50% Workload C)
- What are the time requirements for any/all back-end batch processes (specify peak vs. nominal)?
Pre-requisites for Performance Testing
A stable build of the application which must resemble the Production environment as close to possible.The performance testing environment should not be clubbed with User acceptance testing (UAT) or development environment. This is dangerous as if an UAT or Integration test or other tests are going on in the same environment, then the results obtained from the performance testing may not be reliable. As a best practice it is always advisable to have a separate performance testing environment resembling the production environment as much as possible.
Test conditions
In performance testing, it is often crucial (and often difficult to arrange) for the test conditions to be similar to the expected actual use. This is, however, not entirely possible in actual practice. The reason is that the workloads of production systems have a random nature, and while the test workloads do their best to mimic what may happen in the production environment, it is impossible to exactly replicate this workload variability - except in the most simple system.Loosely-coupled architectural implementations (e.g.: SOA
Service-oriented architecture
In software engineering, a Service-Oriented Architecture is a set of principles and methodologies for designing and developing software in the form of interoperable services. These services are well-defined business functionalities that are built as software components that can be reused for...
) have created additional complexities with performance testing. Enterprise services or assets (that share a common infrastructure
Infrastructure
Infrastructure is basic physical and organizational structures needed for the operation of a society or enterprise, or the services and facilities necessary for an economy to function...
or platform) require coordinated performance testing (with all consumers creating production-like transaction volumes and load on shared infrastructures or platforms) to truly replicate production-like states. Due to the complexity and financial and time requirements around this activity, some organizations now employ tools that can monitor and create production-like conditions (also referred as "noise") in their performance testing environments (PTE
PTE
PTE may refer to:*CVH-PTE Engine, an engine produced by the Ford Motor Company*Partido de los Trabajadores de Ecuador, the Workers' Party of Ecuador*Partido del Trabajo de España, the Party of Labour of Spain...
) to understand capacity and resource requirements and verify / validate quality attributes.
Timing
It is critical to the cost performance of a new system, that performance test efforts begin at the inception of the development project and extend through to deployment. The later a performance defect is detected, the higher the cost of remediation. This is true in the case of functional testing, but even more so with performance testing, due to the end-to-end nature of its scope.Tools
In the diagnostic case, software engineerSoftware engineer
A software engineer is an engineer who applies the principles of software engineering to the design, development, testing, and evaluation of the software and systems that make computers or anything containing software, such as computer chips, work.- Overview :...
s use tools such as profilers to measure what parts of a device or software contributes most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time.
Myths of Performance Testing
Some of the very common myths are given below.1. Performance Testing is done to break the system.
Stress Testing is done to understand the break point of the system. Otherwise normal load testing is generally done to understand the behavior of the application under the expected user load. Depending on other requirements, such as expectation of spike load, continued load for an extended period of time would demand spike, endurance soak or stress testing.
2. Performance Testing should only be done after the System Integration Testing
Although this is mostly the norm in the industry, performance testing can also be done while the initial development of the application is taking place. This kind of approach is known as the Early Performance Testing. This approach would ensure a holistic development of the application keeping the performance parameters in mind. Thus the finding of a performance bug just before the release of the application and the cost involved in rectifying the bug is reduced to a great extent.
3. Performance Testing only involves creation of scripts and any application changes would cause a simple refactoring of the scripts.
Performance Testing in itself is an evolving science in the Software Industry
Software industry
The software industry includes businesses involved in the development, maintenance and publication of computer software using any business model...
. Scripting itself although important, is only one of the components of the performance testing. The major challenge for any performance tester is to determine the type of tests needed to execute and analyzing the various performance counters to determine the performance bottleneck.
The other segment of the myth concerning the change in application would result only in little refactoring in the scripts is also untrue as any form of change on the UI especially in the Web protocol would entail complete re-development of the scripts from scratch. This problem becomes bigger if the protocols involved include Web Services, Siebel
Siebel
Siebel, originally Flugzeugbau Halle, was a German aircraft manufacturer founded in 1937 in Halle an der Saale. It was revived as Siebelwerke in 1948. In 1952, it merged with ATG to form Siebelwerke-ATG Gmbh...
, Citrix, and SAP
SAP ERP
The SAP ERP application is an integrated enterprise resource planning software manufactured by SAP AG that targets business software requirements of midsize and large organizations in all industries and sectors...
.
Technology
Performance testing technology employs one or more PCs or Unix servers to act as injectors – each emulating the presence of numbers of users and each running an automated sequence of interactions (recorded as a script, or as a series of scripts to emulate different types of user interaction) with the host whose performance is being tested. Usually, a separate PC acts as a test conductor, coordinating and gathering metrics from each of the injectors and collating performance data for reporting purposes. The usual sequence is to ramp up the load – starting with a small number of virtual users and increasing the number over a period to some maximum. The test result shows how the performance varies with the load, given as number of users vs response time. Various tools, are available to perform such tests. Tools in this category usually execute a suite of tests which will emulate real users against the system. Sometimes the results can reveal oddities, e.g., that while the average response time might be acceptable, there are outliers of a few key transactions that take considerably longer to complete – something that might be caused by inefficient database queries, pictures etc.Performance testing can be combined with stress testing
Stress testing (software)
In software testing, stress testing refers to tests that determine the robustness of software by testing beyond the limits of normal operation. Stress testing is particularly important for "mission critical" software, but is used for all types of software...
, in order to see what happens when an acceptable load is exceeded –does the system crash? How long does it take to recover if a large load is reduced? Does it fail in a way that causes collateral damage?
Analytical Performance Modeling is a method to model the behaviour of an application in a spreadsheet. The model is fed with measurements of transaction resource demands (CPU, disk I/O, LAN
Län
Län and lääni refer to the administrative divisions used in Sweden and previously in Finland. The provinces of Finland were abolished on January 1, 2010....
, WAN
WAN
WAN may refer to:* Wide area network * World Association of Newspapers* Wanborough railway station in England, which has the station code WAN...
), weighted by the transaction-mix (business transactions per hour). The weighted transaction resource demands are added-up to obtain the hourly resource demands and divided by the hourly resource capacity to obtain the resource loads. Using the responsetime formula (R=S/(1-U), R=responsetime, S=servicetime, U=load), responsetimes can be calculated and calibrated with the results of the performance tests. Analytical performance modelling allows evaluation of design options and system sizing based on actual or anticipated business usage. It is therefore much faster and cheaper than performance testing, though it requires thorough understanding of the hardware platforms.
Tasks to undertake
Tasks to perform such a test would include:- Decide whether to use internal or external resources to perform the tests, depending on inhouse expertise (or lack thereof)
- Gather or elicit performance requirements (specifications) from users and/or business analysts
- Develop a high-level planPlanA plan is typically any diagram or list of steps with timing and resources, used to achieve an objective. See also strategy. It is commonly understood as a temporal set of intended actions, through which one expects to achieve a goal...
(or project charter), including requirements, resources, timelines and milestones - Develop a detailed performance test planTest planA test plan is a document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be.-Test plans:...
(including detailed scenarios and test caseTest caseA test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
s, workloads, environment info, etc.) - Choose test tool(s)
- Specify test data needed and charter effort (often overlooked, but often the death of a valid performance test)
- Develop proof-of-concept scripts for each application/component under test, using chosen test tools and strategies
- Develop detailed performance test project plan, including all dependencies and associated timelines
- Install and configure injectors/controller
- Configure the test environment (ideally identical hardware to the production platform), router configuration, quiet network (we don’t want results upset by other users), deployment of server instrumentation, database test sets developed, etc.
- Execute tests – probably repeatedly (iteratively) in order to see whether any unaccounted for factor might affect the results
- Analyze the results - either pass/fail, or investigation of critical path and recommendation of corrective action
Performance Testing Web Applications Methodology
According to the Microsoft Developer Network the Performance Testing Methodology consists of the following activities:- Activity 1. Identify the Test Environment. Identify the physical test environment and the production environment as well as the tools and resources available to the test team. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test designTest designIn software engineering, test design is the act of creating and writing test suites for testing a software.- Definition :Test design could require all or one of:* knowledge of the software, and the business area it operates on,...
and planning and helps you identify testing challenges early in the project. In some situations, this process must be revisited periodically throughout the project’s life cycleLife cycleLife cycle or lifecycle may refer to: * Biological life cycle* Enterprise life cycle* Life cycle assessment* New product development* Product lifecycle , various meanings* Project life cycle...
. - Activity 2. Identify Performance Acceptance Criteria. Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern. Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics.
- Activity 3. Plan and Design Tests. Identify key scenarioScenarioA scenario is a synoptical collage of an event or series of actions and events. In the Commedia dell'arte it was an outline of entrances, exits, and action describing the plot of a play that was literally pinned to the back of the scenery...
s, determine variability among representative users and how to simulate that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed. - Activity 4. Configure the Test Environment. Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary.
- Activity 5. Implement the Test Design. Develop the performance tests in accordance with the test design.
- Activity 6. Execute the Test. Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment.
- Activity 7. Analyze Results, Tune, and Retest. Analyse, Consolidate and share results data. Make a tuning change and retest. Improvement or degradation? Each improvement made will return smaller improvement than the previous improvement. When do you stop? When you reach a CPU bottleneck, the choices then are either improve the code or add more CPU.
See also
- Stress testing (software)Stress testing (software)In software testing, stress testing refers to tests that determine the robustness of software by testing beyond the limits of normal operation. Stress testing is particularly important for "mission critical" software, but is used for all types of software...
- Benchmark (computing)Benchmark (computing)In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it...
- Web server benchmarkingWeb server benchmarkingWeb server benchmarking is the process of estimating a web server performance in order to find if the server can serve sufficiently high workload.- Key parameters :The performance is usually measured in terms of:...
- Application Response MeasurementApplication Response MeasurementApplication Response Measurement is an open standard published by the Open Group for monitoring and diagnosing performance bottlenecks within complex enterprise applications that use loosely-coupled designs or service-oriented architectures....
External links
- The Art of Application Performance Testing - O'Reilly ISBN 978-0-596-52066-3 (Book)
- Performance Testing Guidance for Web Applications (MSDN)
- Performance Testing Guidance for Web Applications (Book)
- Performance Testing Guidance for Web Applications (PDF)
- Performance Testing Guidance (Online KB)
- Enterprise IT Performance Testing (Online KB)
- Performance Testing Videos (MSDN)
- Open Source Performance Testing tools
- "User Experience, not Metrics" and "Beyond Performance Testing"
- "Performance Testing Traps / Pitfalls"