Rate of convergence
Encyclopedia
In numerical analysis
, the speed at which a convergent sequence
approaches its limit is called the rate of convergence. Although strictly speaking, a limit does not give information about any finite first part of the sequence, this concept is of practical importance if we deal with a sequence of successive approximations for an iterative method
, as then typically fewer iterations are needed to yield a useful approximation if the rate of convergence is higher. This may even make the difference between needing ten or a million iterations.
Similar concepts are used for discretization
methods. The solution of the discretized problem converges to the solution of the continuous problem as the grid size goes to zero, and the speed of convergence is one of the factors of the efficiency of the method. However, the terminology in this case is different from the terminology for iterative methods.
Series acceleration
is a collection of techniques for improving the rate of convergence of a series. Such acceleration is commonly accomplished with sequence transformations.
We say that this sequence converges linearly to L, if there exists a number μ ∈ (0, 1) such that
The number μ is called the rate of convergence.
If the sequences converges, and
If the sequences converges sublinearly and additionally
then it is said the sequence {xk} converges logarithmically to L.
The next definition is used to distinguish superlinear rates of convergence. We say that the sequence converges with order q for q > 1 to L if
In particular, convergence with order
This is sometimes called Q-linear convergence, Q-quadratic convergence, etc., to distinguish it from the definition below. The Q stands for "quotient," because the definition uses the quotient between two successive terms.
Under the new definition, the sequence {xk} converges with at least order q if there exists a sequence {εk} such that
and the sequence {εk} converges to zero with order q according to the above "simple" definition. To distinguish it from that definition, this is sometimes called R-linear convergence, R-quadratic convergence, etc. (with the R standing for "root").
The sequence {ak} converges linearly to 0 with rate 1/2. More generally, the sequence Cμk converges linearly with rate μ if |μ| < 1. The sequence {bk} also converges linearly to 0 with rate 1/2 under the extended definition, but not under the simple definition. The sequence {ck} converges superlinearly. In fact, it is quadratically convergent. Finally, the sequence {dk} converges sublinearly.
In this case, a sequence
is said to converge to L with order p if there exists a constant C such that
This is written as |xn - L| = O(n-p) using the big O notation
.
This is the relevant definition when discussing methods for numerical quadrature or the solution of ordinary differential equations
.
The sequence {ak} with ak = 2-k, which was also introduced above, converges with order p for every number p. It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.
i.e. to transform a given sequence into one converging faster to the same limit. Such techniques are in general known as "series acceleration
". The goal of the transformed sequence is to be much less "expensive" to calculate than the original sequence. One example of series acceleration is Aitken's delta-squared process
.
Numerical analysis
Numerical analysis is the study of algorithms that use numerical approximation for the problems of mathematical analysis ....
, the speed at which a convergent sequence
Limit of a sequence
The limit of a sequence is, intuitively, the unique number or point L such that the terms of the sequence become arbitrarily close to L for "large" values of n...
approaches its limit is called the rate of convergence. Although strictly speaking, a limit does not give information about any finite first part of the sequence, this concept is of practical importance if we deal with a sequence of successive approximations for an iterative method
Iterative method
In computational mathematics, an iterative method is a mathematical procedure that generates a sequence of improving approximate solutions for a class of problems. A specific implementation of an iterative method, including the termination criteria, is an algorithm of the iterative method...
, as then typically fewer iterations are needed to yield a useful approximation if the rate of convergence is higher. This may even make the difference between needing ten or a million iterations.
Similar concepts are used for discretization
Discretization
In mathematics, discretization concerns the process of transferring continuous models and equations into discrete counterparts. This process is usually carried out as a first step toward making them suitable for numerical evaluation and implementation on digital computers...
methods. The solution of the discretized problem converges to the solution of the continuous problem as the grid size goes to zero, and the speed of convergence is one of the factors of the efficiency of the method. However, the terminology in this case is different from the terminology for iterative methods.
Series acceleration
Series acceleration
In mathematics, series acceleration is one of a collection of sequence transformations for improving the rate of convergence of a series. Techniques for series acceleration are often applied in numerical analysis, where they are used to improve the speed of numerical integration...
is a collection of techniques for improving the rate of convergence of a series. Such acceleration is commonly accomplished with sequence transformations.
Basic definition
Suppose that the sequence {xk} converges to the number L.We say that this sequence converges linearly to L, if there exists a number μ ∈ (0, 1) such that
The number μ is called the rate of convergence.
If the sequences converges, and
- μ = 0, then the sequence is said to converge superlinearly.
- μ = 1, then the sequence is said to converges sublinearly.
If the sequences converges sublinearly and additionally
then it is said the sequence {xk} converges logarithmically to L.
The next definition is used to distinguish superlinear rates of convergence. We say that the sequence converges with order q for q > 1 to L if
In particular, convergence with order
- 2 is called quadratic convergence,
- 3 is called cubic convergence,
- etc.
This is sometimes called Q-linear convergence, Q-quadratic convergence, etc., to distinguish it from the definition below. The Q stands for "quotient," because the definition uses the quotient between two successive terms.
Extended definition
The drawback of the above definitions is that these do not catch some sequences which still converge reasonably fast, but whose "speed" is variable, such as the sequence {bk} below. Therefore, the definition of rate of convergence is sometimes extended as follows.Under the new definition, the sequence {xk} converges with at least order q if there exists a sequence {εk} such that
and the sequence {εk} converges to zero with order q according to the above "simple" definition. To distinguish it from that definition, this is sometimes called R-linear convergence, R-quadratic convergence, etc. (with the R standing for "root").
Examples
Consider the following sequences:The sequence {ak} converges linearly to 0 with rate 1/2. More generally, the sequence Cμk converges linearly with rate μ if |μ| < 1. The sequence {bk} also converges linearly to 0 with rate 1/2 under the extended definition, but not under the simple definition. The sequence {ck} converges superlinearly. In fact, it is quadratically convergent. Finally, the sequence {dk} converges sublinearly.
Convergence speed for discretization methods
A similar situation exists for discretization methods. Here, the important parameter is not the iteration number k but the number of grid points, here denoted n. In the simplest situation (a uniform one-dimensional grid), the number of grid points is inversely proportional to the grid spacing.In this case, a sequence
is said to converge to L with order p if there exists a constant C such thatThis is written as |xn - L| = O(n-p) using the big O notation
Big O notation
In mathematics, big O notation is used to describe the limiting behavior of a function when the argument tends towards a particular value or infinity, usually in terms of simpler functions. It is a member of a larger family of notations that is called Landau notation, Bachmann-Landau notation, or...
.
This is the relevant definition when discussing methods for numerical quadrature or the solution of ordinary differential equations
Numerical ordinary differential equations
Numerical ordinary differential equations is the part of numerical analysis which studies the numerical solution of ordinary differential equations...
.
Examples
The sequence {dk} with dk = 1 / (k+1) was introduced above. This sequence converges with order 1 according to the convention for discretization methods.The sequence {ak} with ak = 2-k, which was also introduced above, converges with order p for every number p. It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.
Acceleration of convergence
Many methods exist to increase the rate of convergence of a given sequence,i.e. to transform a given sequence into one converging faster to the same limit. Such techniques are in general known as "series acceleration
Series acceleration
In mathematics, series acceleration is one of a collection of sequence transformations for improving the rate of convergence of a series. Techniques for series acceleration are often applied in numerical analysis, where they are used to improve the speed of numerical integration...
". The goal of the transformed sequence is to be much less "expensive" to calculate than the original sequence. One example of series acceleration is Aitken's delta-squared process
Aitken's delta-squared process
In numerical analysis, Aitken's delta-squared process is a series acceleration method, used for accelerating the rate of convergence of a sequence. It is named after Alexander Aitken, who introduced this method in 1926. Its early form was known to Seki Kōwa and was found for rectification of the...
.