Positive-definite matrix
Encyclopedia
In linear algebra
, a positive-definite matrix is a matrix that in many ways is analogous to a positive real number
. The notion is closely related to a positive-definite
symmetric bilinear form (or a sesquilinear form
in the complex case).
The proper definition of positive-definite is unambiguous for Hermitian matrices, but there is no agreement in the literature on how this should be extended for non-Hermitian matrices, if at all. (See the section Non-Hermitian matrices below.)
M is positive definite if zTMz > 0 for all non-zero vectors z with real entries (
), where zT denotes the transpose
of z.
An complex matrix M is positive definite if ℜ(z*Mz) > 0 for all non-zero complex vectors z, where z* denotes the conjugate transpose
of z and ℜ(c) is the real part of a complex number c.
An complex Hermitian matrix M is positive definite if z*Mz > 0 for all non-zero complex vectors z. The quantity z*Mz is always real because M is a Hermitian matrix.
when the entries z0, z1 are real and at least one of them nonzero, this is positive.
It is positive definite since for any non-zero vector
,
we have
which is a sum of squares and therefore nonnegative; in fact, each squared summand can be zero only when
, so 
is indeed positive-definite.
For real symmetric matrices, these properties can be simplified by replacing
with 
, and "conjugate transpose" with "transpose."
s. Let K be the field
R or C, and V be a vector space
over K. A Hermitian form
is a bilinear map such that B(x, y) is always the complex conjugate of B(y, x). Such a function B is called positive definite if B(x, x) > 0 for every nonzero x in V.
matrices. In this section we write for the real case, extension to the complex case is immediate.
Let
and 
be two positive-definite matrices. Write the generalized eigenvalue equation as
where 
. Now we can decompose the inverse of 
as 
(so
must be positive definite, as the proof shows, in fact it is enough that 
is symmetric). Now multiply various places with 
to get
which we can rewrite as
where 
. Manipulation now yields 
where
is a matrix having as columns the generalized eigenvectors and 
is a diagonal matrix with the
generalized eigenvalues. Now premultiplication with
gives the final result:
and 
,
but note that this is no longer an orthogonal diagonalization.
Note that this result does not contradict what is said on simultaneous diagonalization in the article
Diagonalizable matrix#Simultaneous diagonalisation, which refers to simultaneous diagonalization by a similarity transformation. Our result here is more akin to a simultaneous diagonalization of two
quadratic forms, and is useful for optimization of one form under conditions on the other. For this result see Horn&Johnson, 1985, page 218 and following.
for all non-zero
(or, all non-zero 
for the real matrix).
It is called positive-semidefinite (or sometimes nonnegative-definite) if
for all
(or, all 
for the real matrix), where 
is the conjugate transpose of x.
It is called negative-semidefinite if
for all
(or, all 
for the real matrix).
A matrix M is positive-semidefinite if and only if it arises as the Gram matrix of some set of vectors. In contrast to the positive-definite case, these vectors need not be linearly independent.
For any matrix A, the matrix A*A is positive semidefinite, and rank(A) = rank(A*A). Conversely, any Hermitian positive semidefinite matrix M can be written as M = A*A; this is the Cholesky decomposition
.
A Hermitian matrix which is neither positive- nor negative-semidefinite is called indefinite.
A matrix is negative definite if all kth order leading principal minors are negative if k is odd and positive if k is even.
A Hermitian matrix is negative-definite, negative-semidefinite, or positive-semidefinite if and only if all of its eigenvalues are negative, non-positive, or non-negative, respectively.
where positive definite matrices define positive operators.
For arbitrary square matrices M,N we write M ≥ N if M − N ≥ 0; i.e., M − N is positive semi-definite. This defines a partial ordering
on the set of all square matrices. One can similarly define a strict partial ordering M > N.
:
Where each block is n × n. By applying the positivity condition, it immediately follows that A and D are hermitian, and C = B*.
We have that z*Mz ≥ 0 for all complex z, and in particular for z = ( v, 0)T. Then
A similar argument can be applied to D, and thus we conclude that both A and D must be positive definite matrices, as well.
satisfies this property, because for all real vectors
such that 
,
In general, we have xTMx > 0 for all real nonzero vectors x if and only if the symmetric part, (M + MT) / 2, is positive definite.
The situation for complex matrices may be different, depending on how one generalizes the inequality z*Mz > 0 when considering M which aren't necessarily Hermitian. If z*Mz is real for all complex vectors z, then the matrix M must be Hermitian. To see this, we define the Hermitian matrices A=(M+M*)/2 and B=(M-M*)/(2i) so that M=A+iB. Then, z*Mz=z*Az+iz*Bz is real. By the Hermiticity of A and B, z*Az and z*Bz are individually real so z*Bz must be zero for all z. So then B is the zero matrix and M=A, which is Hermitian.
So, if we require that z*Mz be real and positive, then M is automatically Hermitian. On the other hand, we have that Re(z*Mz) > 0 for all complex nonzero vectors z if and only if the Hermitian part, (M + M*) / 2, is positive definite.
In summary, the distinguishing feature between the real and complex case is that, a bounded
positive operator on a complex Hilbert space is necessarily Hermitian, or self adjoint. The general claim can be argued using the polarization identity
. That is no longer true in the real case.
There is no agreement in the literature on the proper definition of positive-definite for non-Hermitian matrices.
Linear algebra
Linear algebra is a branch of mathematics that studies vector spaces, also called linear spaces, along with linear functions that input one vector and output another. Such functions are called linear maps and can be represented by matrices if a basis is given. Thus matrix theory is often...
, a positive-definite matrix is a matrix that in many ways is analogous to a positive real number
Real number
In mathematics, a real number is a value that represents a quantity along a continuum, such as -5 , 4/3 , 8.6 , √2 and π...
. The notion is closely related to a positive-definite
Definite bilinear form
In mathematics, a definite bilinear form is a bilinear form B over some vector space V such that the associated quadratic formQ=B \,...
symmetric bilinear form (or a sesquilinear form
Sesquilinear form
In mathematics, a sesquilinear form on a complex vector space V is a map V × V → C that is linear in one argument and antilinear in the other. The name originates from the numerical prefix sesqui- meaning "one and a half"...
in the complex case).
The proper definition of positive-definite is unambiguous for Hermitian matrices, but there is no agreement in the literature on how this should be extended for non-Hermitian matrices, if at all. (See the section Non-Hermitian matrices below.)
Definition
An real matrixMatrix (mathematics)
In mathematics, a matrix is a rectangular array of numbers, symbols, or expressions. The individual items in a matrix are called its elements or entries. An example of a matrix with six elements isMatrices of the same size can be added or subtracted element by element...
M is positive definite if zTMz > 0 for all non-zero vectors z with real entries (
), where zT denotes the transposeTranspose
In linear algebra, the transpose of a matrix A is another matrix AT created by any one of the following equivalent actions:...
of z.
An complex matrix M is positive definite if ℜ(z*Mz) > 0 for all non-zero complex vectors z, where z* denotes the conjugate transpose
Conjugate transpose
In mathematics, the conjugate transpose, Hermitian transpose, Hermitian conjugate, or adjoint matrix of an m-by-n matrix A with complex entries is the n-by-m matrix A* obtained from A by taking the transpose and then taking the complex conjugate of each entry...
of z and ℜ(c) is the real part of a complex number c.
An complex Hermitian matrix M is positive definite if z*Mz > 0 for all non-zero complex vectors z. The quantity z*Mz is always real because M is a Hermitian matrix.
Examples
- The matrix
is positive definite. For a vector with entries 
the quadratic form is 
when the entries z0, z1 are real and at least one of them nonzero, this is positive.
- An example of a matrix that is not positive, but is positive-definite, is given by
It is positive definite since for any non-zero vector
,we have
which is a sum of squares and therefore nonnegative; in fact, each squared summand can be zero only when
, so is indeed positive-definite.- Conversely, the strictly positive matrix
is not positive definite, showing (together with the previous example) that these two properties are independent. Evaluated at 
, the quadratic form is 
- For a huge class of examples, consider that in statisticsStatisticsStatistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
positive definite matrices appear as covariance matricesCovariance matrixIn probability theory and statistics, a covariance matrix is a matrix whose element in the i, j position is the covariance between the i th and j th elements of a random vector...
. In fact, every positive definite matrix is the covariance matrix of some multivariate probability distribution.
Characterizations
Let M be an n × n Hermitian matrix. The following properties are equivalent to M being positive definite:| 1. | All eigenvalues λi of M are positive. Recall that any Hermitian M has an eigendecomposition Eigendecomposition of a matrix In the mathematical discipline of linear algebra, eigendecomposition or sometimes spectral decomposition is the factorization of a matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors... M = P−1DP where P is a unitary matrix whose rows are orthonormal eigenvectors of M, forming a basis, and D is a diagonal matrix Diagonal matrix In linear algebra, a diagonal matrix is a matrix in which the entries outside the main diagonal are all zero. The diagonal entries themselves may or may not be zero... . Therefore M may be regarded as a real diagonal matrix D that has been re-expressed in some new coordinate system. This characterization means that M is positive definite if and only if the diagonal elements of D (the eigenvalues) are all positive. In other words, in the basis consisting of the eigenvectors of M, the action of M is component-wise multiplication with a (fixed) element in Cn with positive entries. |
| 2. | The sesquilinear form Sesquilinear form In mathematics, a sesquilinear form on a complex vector space V is a map V × V → C that is linear in one argument and antilinear in the other. The name originates from the numerical prefix sesqui- meaning "one and a half"...  defines an inner product on Cn. (In fact, every inner product on Cn arises in this fashion from a Hermitian positive definite matrix.) In particular, positive definiteness for a Hermitian M is equivalent to the fact that  for all nonzero x. |
| 3. | M is the Gram matrix of some collection of linearly independent vectors for some k. That is, M satisfies:  The vectors xi may optionally be restricted to fall in Cn. In other words, M is of the form  where A is not necessarily square but must be injective in general. |
| 4. | All the following matrices have a positive determinant Determinant In linear algebra, the determinant is a value associated with a square matrix. It can be computed from the entries of the matrix by a specific arithmetic expression, while other ways to determine its value exist as well... (Sylvester's criterion Sylvester's criterion In mathematics, Sylvester’s criterion is a necessary and sufficient criterion to determine whether a Hermitian matrix is positive-definite. It is named after James Joseph Sylvester.... ):
In other words, all of the leading principal minors are positive. For positive semidefinite matrices, all principal minors have to be non-negative. The leading principal minors alone do not imply positive semidefiniteness, as can be seen from the example  |
| 5. | There exists a unique lower triangular matrix  , with strictly positive diagonal elements, that allows the factorization of  into .where  is the conjugate transposeConjugate transpose In mathematics, the conjugate transpose, Hermitian transpose, Hermitian conjugate, or adjoint matrix of an m-by-n matrix A with complex entries is the n-by-m matrix A* obtained from A by taking the transpose and then taking the complex conjugate of each entry... of  .This factorization is called Cholesky decomposition Cholesky decomposition In linear algebra, the Cholesky decomposition or Cholesky triangle is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose. It was discovered by André-Louis Cholesky for real matrices... . |
| 6. | The quadratic function Quadratic function A quadratic function, in mathematics, is a polynomial function of the formf=ax^2+bx+c,\quad a \ne 0.The graph of a quadratic function is a parabola whose axis of symmetry is parallel to the y-axis.... associated with M  is, regardless of b, a strictly convex function.  has a unique global minimum, and this is why positive definite matrices are so common in optimizationOptimization (mathematics) In mathematics, computational science, or management science, mathematical optimization refers to the selection of a best element from some set of available alternatives.... problems. |
itselfFor real symmetric matrices, these properties can be simplified by replacing
with , and "conjugate transpose" with "transpose."Quadratic forms
Echoing condition 2 above, one can also formulate positive-definiteness in terms of quadratic formQuadratic form
In mathematics, a quadratic form is a homogeneous polynomial of degree two in a number of variables. For example,4x^2 + 2xy - 3y^2\,\!is a quadratic form in the variables x and y....
s. Let K be the field
Field (mathematics)
In abstract algebra, a field is a commutative ring whose nonzero elements form a group under multiplication. As such it is an algebraic structure with notions of addition, subtraction, multiplication, and division, satisfying certain axioms...
R or C, and V be a vector space
Vector space
A vector space is a mathematical structure formed by a collection of vectors: objects that may be added together and multiplied by numbers, called scalars in this context. Scalars are often taken to be real numbers, but one may also consider vector spaces with scalar multiplication by complex...
over K. A Hermitian form
is a bilinear map such that B(x, y) is always the complex conjugate of B(y, x). Such a function B is called positive definite if B(x, x) > 0 for every nonzero x in V.
Simultaneous diagonalization
Two symmetric, positive-definite matrices can be simultaneously diagonalized, although not necessarily via a similarity transformation. This result does not extend to the case of three or morematrices. In this section we write for the real case, extension to the complex case is immediate.
Let
and be two positive-definite matrices. Write the generalized eigenvalue equation as where . Now we can decompose the inverse of as (so
must be positive definite, as the proof shows, in fact it is enough that is symmetric). Now multiply various places with to get which we can rewrite as where . Manipulation now yields where is a matrix having as columns the generalized eigenvectors and is a diagonal matrix with thegeneralized eigenvalues. Now premultiplication with
gives the final result: and ,but note that this is no longer an orthogonal diagonalization.
Note that this result does not contradict what is said on simultaneous diagonalization in the article
Diagonalizable matrix#Simultaneous diagonalisation, which refers to simultaneous diagonalization by a similarity transformation. Our result here is more akin to a simultaneous diagonalization of two
quadratic forms, and is useful for optimization of one form under conditions on the other. For this result see Horn&Johnson, 1985, page 218 and following.
Negative-definite, semidefinite and indefinite matrices
The Hermitian matrix M is said to be negative-definite iffor all non-zero
(or, all non-zero for the real matrix).It is called positive-semidefinite (or sometimes nonnegative-definite) if
for all
(or, all for the real matrix), where is the conjugate transpose of x.It is called negative-semidefinite if
for all
(or, all for the real matrix).A matrix M is positive-semidefinite if and only if it arises as the Gram matrix of some set of vectors. In contrast to the positive-definite case, these vectors need not be linearly independent.
For any matrix A, the matrix A*A is positive semidefinite, and rank(A) = rank(A*A). Conversely, any Hermitian positive semidefinite matrix M can be written as M = A*A; this is the Cholesky decomposition
Cholesky decomposition
In linear algebra, the Cholesky decomposition or Cholesky triangle is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose. It was discovered by André-Louis Cholesky for real matrices...
.
A Hermitian matrix which is neither positive- nor negative-semidefinite is called indefinite.
A matrix is negative definite if all kth order leading principal minors are negative if k is odd and positive if k is even.
A Hermitian matrix is negative-definite, negative-semidefinite, or positive-semidefinite if and only if all of its eigenvalues are negative, non-positive, or non-negative, respectively.
Further properties
If M is an Hermitian positive-semidefinite matrix, one sometimes writes M ≥ 0 and if M is positive-definite one writes M > 0. The notion comes from functional analysisFunctional analysis
Functional analysis is a branch of mathematical analysis, the core of which is formed by the study of vector spaces endowed with some kind of limit-related structure and the linear operators acting upon these spaces and respecting these structures in a suitable sense...
where positive definite matrices define positive operators.
For arbitrary square matrices M,N we write M ≥ N if M − N ≥ 0; i.e., M − N is positive semi-definite. This defines a partial ordering
Partially ordered set
In mathematics, especially order theory, a partially ordered set formalizes and generalizes the intuitive concept of an ordering, sequencing, or arrangement of the elements of a set. A poset consists of a set together with a binary relation that indicates that, for certain pairs of elements in the...
on the set of all square matrices. One can similarly define a strict partial ordering M > N.
- Every positive definite matrix is invertible and its inverse is also positive definite. If M ≥ N > 0 then N−1 ≥ M−1 > 0.
- If M is positive definite and r > 0 is a real number, then r M is positive definite.
If M and N are positive definite, then the sum M + N and the products MNM and NMN are also positive definite. If MN = NM, then MN is also positive definite.
- If M = (mij) ≥ 0 then the diagonal entries mii are real and non-negative. As a consequence the traceTrace (linear algebra)In linear algebra, the trace of an n-by-n square matrix A is defined to be the sum of the elements on the main diagonal of A, i.e.,...
, tr(M) ≥ 0. Furthermore
- and thus
- and thus
- If
is a symmetric matrix of the form 
, and the strict inequality holds
then
is strictly positive definite.
- A matrix M is positive semi-definite if and only if there is a positive semi-definite matrix B with B2 = M. This matrix B is unique, is called the square rootSquare root of a matrixIn mathematics, the square root of a matrix extends the notion of square root from numbers to matrices. A matrix B is said to be a square root of A if the matrix product B · B is equal to A.-Properties:...
of M, and is denoted with B = M1/2 (the square root B is not to be confused with the matrix L in the Cholesky factorization M = LL*, which is also sometimes called the square root of M). If M > N > 0 then M1/2 > N1/2 > 0.
- If M,N > 0 then M ⊗ N > 0, where ⊗ denotes Kronecker productKronecker productIn mathematics, the Kronecker product, denoted by ⊗, is an operation on two matrices of arbitrary size resulting in a block matrix. It gives the matrix of the tensor product with respect to a standard choice of basis. The Kronecker product should not be confused with the usual matrix...
.
- For matrices M = (mij) and N = (nij), write M ° N for the entry-wise product of M and N; i.e., the matrix whose i,j entry is mijnij. Then M ° N is the Hadamard product of M and N. The Hadamard product of two positive-definite matrices is again positive-definite and the Hadamard product of two positive-semidefinite matrices is again positive-semidefinite (this result is often called the Schur product theorem). Furthermore, if M and N are positive-semidefinite, then the following inequality, due to Oppenheim, holds:
- Let M > 0 and N Hermitian. If MN + NM ≥ 0 (resp., MN + NM > 0) then N ≥ 0 (resp., N > 0).
- If M,N ≥ 0 are real positive semidefinite symmetric matrices, then
(Lancaster-Tismenetsky, The Theory of Matrices, p. 218).
- If M > 0 is real, then there is a δ > 0 such that M > δI, where I is the identity matrixIdentity matrixIn linear algebra, the identity matrix or unit matrix of size n is the n×n square matrix with ones on the main diagonal and zeros elsewhere. It is denoted by In, or simply by I if the size is immaterial or can be trivially determined by the context...
.
Block Matrices
A positive 2n × 2n matrix may also be defined by blocksBlock matrix
In the mathematical discipline of matrix theory, a block matrix or a partitioned matrix is a matrix broken into sections called blocks. Looking at it another way, the matrix is written in terms of smaller matrices. We group the rows and columns into adjacent 'bunches'. A partition is the rectangle...
:
Where each block is n × n. By applying the positivity condition, it immediately follows that A and D are hermitian, and C = B*.
We have that z*Mz ≥ 0 for all complex z, and in particular for z = ( v, 0)T. Then
A similar argument can be applied to D, and thus we conclude that both A and D must be positive definite matrices, as well.
Non-Hermitian matrices
A real matrix M may have the property that xTMx > 0 for all nonzero real vectors x without being symmetric. The matrixsatisfies this property, because for all real vectors
such that ,In general, we have xTMx > 0 for all real nonzero vectors x if and only if the symmetric part, (M + MT) / 2, is positive definite.
The situation for complex matrices may be different, depending on how one generalizes the inequality z*Mz > 0 when considering M which aren't necessarily Hermitian. If z*Mz is real for all complex vectors z, then the matrix M must be Hermitian. To see this, we define the Hermitian matrices A=(M+M*)/2 and B=(M-M*)/(2i) so that M=A+iB. Then, z*Mz=z*Az+iz*Bz is real. By the Hermiticity of A and B, z*Az and z*Bz are individually real so z*Bz must be zero for all z. So then B is the zero matrix and M=A, which is Hermitian.
So, if we require that z*Mz be real and positive, then M is automatically Hermitian. On the other hand, we have that Re(z*Mz) > 0 for all complex nonzero vectors z if and only if the Hermitian part, (M + M*) / 2, is positive definite.
In summary, the distinguishing feature between the real and complex case is that, a bounded
Bounded operator
In functional analysis, a branch of mathematics, a bounded linear operator is a linear transformation L between normed vector spaces X and Y for which the ratio of the norm of L to that of v is bounded by the same number, over all non-zero vectors v in X...
positive operator on a complex Hilbert space is necessarily Hermitian, or self adjoint. The general claim can be argued using the polarization identity
Polarization identity
In mathematics, the polarization identity is any one of a family of formulas that express the inner product of two vectors in terms of the norm of a normed vector space. Let \|x\| \, denote the norm of vector x and \langle x, \ y \rangle \, the inner product of vectors x and y...
. That is no longer true in the real case.
There is no agreement in the literature on the proper definition of positive-definite for non-Hermitian matrices.
See also
- Cholesky decompositionCholesky decompositionIn linear algebra, the Cholesky decomposition or Cholesky triangle is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose. It was discovered by André-Louis Cholesky for real matrices...
- Positive-definite function
- Positive definite kernel
- Schur complementSchur complementIn linear algebra and the theory of matrices,the Schur complement of a matrix block is defined as follows.Suppose A, B, C, D are respectivelyp×p, p×q, q×p...
- Square root of a matrixSquare root of a matrixIn mathematics, the square root of a matrix extends the notion of square root from numbers to matrices. A matrix B is said to be a square root of A if the matrix product B · B is equal to A.-Properties:...
- Sylvester's criterionSylvester's criterionIn mathematics, Sylvester’s criterion is a necessary and sufficient criterion to determine whether a Hermitian matrix is positive-definite. It is named after James Joseph Sylvester....
- Covariance matrixCovariance matrixIn probability theory and statistics, a covariance matrix is a matrix whose element in the i, j position is the covariance between the i th and j th elements of a random vector...