Master theorem
Encyclopedia
In the analysis of algorithms
, the master theorem provides a cookbook solution in asymptotic terms (using Big O notation
) for recurrence relation
s of types that occur in the analysis of many divide and conquer algorithm
s. It was popularized by the canonical algorithms textbook Introduction to Algorithms
by Cormen
, Leiserson
, Rivest
, and Stein
, which introduces and proves it in Sections 4.5 and 4.6 (Chapter 4, Third Edition), respectively. Nevertheless, not all recurrence relations can be solved with the use of the master theorem; its generalizations include the Akra–Bazzi method.
In the above algorithm we are dividing the problem into a number of sub problems recursively, each sub problem being of size n/b. This can be visualized as building a call tree with each node of the tree as an instance of one recursive call and its child nodes being instances of subsequent calls. In the above example, each node would have a number of child nodes. Each node does an amount of work that corresponds to the size of the sub problem n passed to that instance of the recursive call and given by
. For example, if each recursive call is doing a comparison sort
, then the amount of work done by each node in the tree is at least O(n log n). The total amount of work done by the entire tree is the sum of the work performed by all the nodes in the tree.
Algorithm such as above can be represented as recurrence relationship
. This recursive relationship can be successively substituted in to itself and expanded to obtain expression for total amount of work done.
The original Master theorem allows to easily calculate run time of such a recursive algorithm in Big O notation
without doing expansion of above recursive relationship. A generalized form of Master Theorem by Akra and Bazzi
introduced in 1998 is applicable on wide number of cases that occur in practice.
In the application to the analysis of a recursive algorithm, the constants and function take on the following significance:
It is possible to determine an asymptotic tight bound in these three cases:
for some constant 
(using Big O notation
)
it follows that:
As one can see in the formula above, the variables get the following values:
Now we have to check that the following equation holds:
If we choose
= 1, we get:
Since this equation holds, the first case of the master theorem applies to the given recurrence relation, thus resulting in the conclusion:
If we insert the values from above, we finally get:
Thus the given recurrence relation T(n) was in Θ(n3).
(This result is confirmed by the exact solution of the recurrence relation, which is
, assuming 
).
it follows that:
As we can see in the formula above the variables get the following values:
Now we have to check that the following equation holds (in this case k=0):
If we insert the values from above, we get:
Since this equation holds, the second case of the master theorem applies to the given recurrence relation, thus resulting in the conclusion:
If we insert the values from above, we finally get:
Thus the given recurrence relation T(n) was in Θ(n log n).
(This result is confirmed by the exact solution of the recurrence relation, which is
, assuming 
.)
for some constant 
and if it is also true that:
for some constant 
and sufficiently large n
it follows that:
As we can see in the formula above the variables get the following values:
Now we have to check that the following equation holds:
If we insert the values from above, and choose
= 1, we get:
Since this equation holds, we have to check the second condition, namely if it is true that:
If we insert once more the values from above, we get the number :
If we choose
, it is true that:
So it follows:
If we insert once more the necessary values, we get:
Thus the given recurrence relation T(n) was in Θ(n2), that complies with the f (n) of the original formula.
(This result is confirmed by the exact solution of the recurrence relation, which is
, assuming 
.)
In the second inadmissible example above, the difference between
and 
can be expressed with the ratio 
. It is clear that 
for any constant 
. Therefore, the difference is not polynomial and the Master Theorem does not apply.
Analysis of algorithms
To analyze an algorithm is to determine the amount of resources necessary to execute it. Most algorithms are designed to work with inputs of arbitrary length...
, the master theorem provides a cookbook solution in asymptotic terms (using Big O notation
Big O notation
In mathematics, big O notation is used to describe the limiting behavior of a function when the argument tends towards a particular value or infinity, usually in terms of simpler functions. It is a member of a larger family of notations that is called Landau notation, Bachmann-Landau notation, or...
) for recurrence relation
Recurrence relation
In mathematics, a recurrence relation is an equation that recursively defines a sequence, once one or more initial terms are given: each further term of the sequence is defined as a function of the preceding terms....
s of types that occur in the analysis of many divide and conquer algorithm
Divide and conquer algorithm
In computer science, divide and conquer is an important algorithm design paradigm based on multi-branched recursion. A divide and conquer algorithm works by recursively breaking down a problem into two or more sub-problems of the same type, until these become simple enough to be solved directly...
s. It was popularized by the canonical algorithms textbook Introduction to Algorithms
Introduction to Algorithms
Introduction to Algorithms is a book by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. It is used as the textbook for algorithms courses at many universities. It is also one of the most commonly cited references for algorithms in published papers, with over 4600...
by Cormen
Thomas H. Cormen
Thomas H. Cormen is the co-author of Introduction to Algorithms, along with Charles Leiserson, Ron Rivest, and Cliff Stein. He is a Full Professor of computer science at Dartmouth College and currently Chair of the Dartmouth College Department of Computer Science. Between 2004 and 2008 he directed...
, Leiserson
Charles E. Leiserson
Charles Eric Leiserson is a computer scientist, specializing in the theory of parallel computing and distributed computing, and particularly practical applications thereof; as part of this effort, he developed the Cilk multithreaded language...
, Rivest
Ron Rivest
Ronald Linn Rivest is a cryptographer. He is the Andrew and Erna Viterbi Professor of Computer Science at MIT's Department of Electrical Engineering and Computer Science and a member of MIT's Computer Science and Artificial Intelligence Laboratory...
, and Stein
Clifford Stein
Clifford Stein, a computer scientist, is currently a professor of industrial engineering and operations research at Columbia University in New York, NY, where he also holds an appointment in the Department of Computer Science. Stein is chair of the Industrial Engineering and Operations Research...
, which introduces and proves it in Sections 4.5 and 4.6 (Chapter 4, Third Edition), respectively. Nevertheless, not all recurrence relations can be solved with the use of the master theorem; its generalizations include the Akra–Bazzi method.
Introduction
Consider a problem that can be solved using a recursive algorithm such as the following:
procedure T( n : size of problem ) defined as:
if n < 1 then exit
Do work of amount f(n)
T(n/b)
T(n/b)
...repeat for a total of a times...
T(n/b)
end procedure
In the above algorithm we are dividing the problem into a number of sub problems recursively, each sub problem being of size n/b. This can be visualized as building a call tree with each node of the tree as an instance of one recursive call and its child nodes being instances of subsequent calls. In the above example, each node would have a number of child nodes. Each node does an amount of work that corresponds to the size of the sub problem n passed to that instance of the recursive call and given by
. For example, if each recursive call is doing a comparison sortComparison sort
A comparison sort is a type of sorting algorithm that only reads the list elements through a single abstract comparison operation that determines which of two elements should occur first in the final sorted list...
, then the amount of work done by each node in the tree is at least O(n log n). The total amount of work done by the entire tree is the sum of the work performed by all the nodes in the tree.
Algorithm such as above can be represented as recurrence relationship
. This recursive relationship can be successively substituted in to itself and expanded to obtain expression for total amount of work done.The original Master theorem allows to easily calculate run time of such a recursive algorithm in Big O notation
Big O notation
In mathematics, big O notation is used to describe the limiting behavior of a function when the argument tends towards a particular value or infinity, usually in terms of simpler functions. It is a member of a larger family of notations that is called Landau notation, Bachmann-Landau notation, or...
without doing expansion of above recursive relationship. A generalized form of Master Theorem by Akra and Bazzi
Akra-Bazzi method
In computer science, the Akra–Bazzi method, or Akra–Bazzi theorem, is used to analyze the asymptotic behavior of the mathematical recurrences that appear in the analysis of divide and conquer algorithms where the sub-problems have substantially different sizes...
introduced in 1998 is applicable on wide number of cases that occur in practice.
Generic form
The master theorem concerns recurrence relations of the form:In the application to the analysis of a recursive algorithm, the constants and function take on the following significance:
- n is the size of the problem.
- a is the number of subproblems in the recursion.
- n/b is the size of each subproblem. (Here it is assumed that all subproblems are essentially the same size.)
- f (n) is the cost of the work done outside the recursive calls, which includes the cost of dividing the problem and the cost of merging the solutions to the subproblems.
It is possible to determine an asymptotic tight bound in these three cases:
Generic form
If for some constant (using Big O notationBig O notation
In mathematics, big O notation is used to describe the limiting behavior of a function when the argument tends towards a particular value or infinity, usually in terms of simpler functions. It is a member of a larger family of notations that is called Landau notation, Bachmann-Landau notation, or...
)
it follows that:
Example
As one can see in the formula above, the variables get the following values:
Now we have to check that the following equation holds:
If we choose
= 1, we get:Since this equation holds, the first case of the master theorem applies to the given recurrence relation, thus resulting in the conclusion:
If we insert the values from above, we finally get:
Thus the given recurrence relation T(n) was in Θ(n3).
(This result is confirmed by the exact solution of the recurrence relation, which is
, assuming ).Generic form
If it is true, for some constant k ≥ 0, that:it follows that:
Example
As we can see in the formula above the variables get the following values:
Now we have to check that the following equation holds (in this case k=0):
If we insert the values from above, we get:
Since this equation holds, the second case of the master theorem applies to the given recurrence relation, thus resulting in the conclusion:
If we insert the values from above, we finally get:
Thus the given recurrence relation T(n) was in Θ(n log n).
(This result is confirmed by the exact solution of the recurrence relation, which is
, assuming .)Generic form
If it is true that: for some constant and if it is also true that:
for some constant and sufficiently large nit follows that:
Example
As we can see in the formula above the variables get the following values:
Now we have to check that the following equation holds:
If we insert the values from above, and choose
= 1, we get:Since this equation holds, we have to check the second condition, namely if it is true that:
If we insert once more the values from above, we get the number :
If we choose
, it is true that: So it follows:
If we insert once more the necessary values, we get:
Thus the given recurrence relation T(n) was in Θ(n2), that complies with the f (n) of the original formula.
(This result is confirmed by the exact solution of the recurrence relation, which is
, assuming .)Inadmissible equations
The following equations cannot be solved using the master theorem:
- a is not a constant
- non-polynomial difference between f(n) and
(see below)
- non-polynomial difference between f(n) and
- a<1 cannot have less than one sub problem
- f(n) is not positive
- case 3 but regularity violation.
In the second inadmissible example above, the difference between
and can be expressed with the ratio . It is clear that for any constant . Therefore, the difference is not polynomial and the Master Theorem does not apply.Application to common algorithms
| Algorithm | Recurrence Relationship | Run time | Comment |
|---|---|---|---|
| Binary search |  |
 |
Apply Master theorem where  |
| Binary tree traversal |  |
 |
Apply Master theorem where  |
| Optimal Sorted Matrix Search |  |
 |
Apply Akra-Bazzi theorem for  and  to get  |
| Merge Sort Merge sort Merge sort is an O comparison-based sorting algorithm. Most implementations produce a stable sort, meaning that the implementation preserves the input order of equal elements in the sorted output. It is a divide and conquer algorithm... |
 |
 |