Tensor product
Encyclopedia
In mathematics
, the tensor product, denoted by ⊗, may be applied in different contexts to vector
s, matrices
, tensor
s, vector space
s, algebras
, topological vector space
s, and module
s, among many other structures or objects. In each case the significance of the symbol is the same: the most general bilinear operation
. In some contexts, this product is also referred to as outer product
. The term "tensor product" is also used in relation to monoidal categories
.
s V and W over a field
K can be defined by the method of generators and relations.
To construct V ⊗ W, one begins with the set of ordered pairs in the Cartesian product
V × W. For the purposes of this construction, regard this Cartesian product as a set rather than a vector space. The free vector space F on V × W is defined by taking the vector space in which the elements of V × W are a basis
. In set-builder notation
,
where we have used the symbol e(v,w) to emphasize that these are taken to be linearly independent
by definition for distinct (v, w) ∈ V × W.
The tensor product arises by defining the following four equivalence relations in F(V × W):
where v1, v2, and w1, w2 are vectors from V and W (respectively), and c is from the underlying field K. Denoting by R the space generated by these four equivalence relations, the tensor product of the two vector spaces V and W is then the quotient space
It is also called the tensor product space of V and W and is a vector space (which can be verified by directly checking the vector space axioms). The tensor product of two elements v and w is the equivalence class (e(v,w) + R) of e(v,w) in V ⊗ W, denoted v ⊗ w. This notation can somewhat obscure the fact that tensors are always coset
s: manipulations performed via the representatives (v,w) must always be checked that they do not depend on the particular choice of representative.
The space R is mapped to zero in V ⊗ W, so that the above three equivalence relations become equalities in the tensor product space:
Given bases {vi} and {wi} for V and W respectively, the tensors {vi ⊗ wj} form a basis for V ⊗ W (generally ordered so that vi ⊗ wj+1 comes before vi+1 ⊗ wj). The dimension of the tensor product therefore is the product of dimensions of the original spaces; for instance Rm ⊗ Rn will have dimension mn.
Elements of V ⊗ W are sometimes referred to as tensors, although this term refers to many other related concepts as well. An element of V ⊗ W of the form v ⊗ w is called a pure or simple tensor. In general, an element of the tensor product space is not a pure tensor, but rather a finite linear combination of pure tensors. That is to say, if v1 and v2 are linearly independent, and w1 and w2 are also linearly independent, then v1 ⊗ w1 + v2 ⊗ w2 cannot be written as a pure tensor. The number of simple tensors required to express an element of a tensor product is called the tensor rank, (not to be confused with tensor order, which is the number of spaces one has taken the product of, in this case 2; in notation, the number of indices) and for linear operators or matrices, thought of as (1,1) tensors (elements of the space V ⊗ V*), it agrees with matrix rank.
Characterization by a universal property
The tensor product of V and W can be defined (up to isomorphism) by any pair (L, φ), with L a vector space on K and 
a bilinear map such that
In this sense, φ is the most general bilinear map that can be built from VxW.
It is easy to check that
satisfies this universal property.
As a functor
The tensor product also operates on linear maps between vector spaces. Specifically, given two linear maps S : V → X and T : W → Y between vector spaces, the tensor product of the two linear maps S and T is a linear map
defined by
In this way, the tensor product becomes a bifunctor from the category of vector spaces to itself, covariant in both arguments.
The Kronecker product of two matrices is the matrix of the tensor product of the two corresponding linear maps under a standard choice of bases of the tensor products (see the article on Kronecker products).
so that, any trilinear map F from the direct product to a vector space W
factors uniquely as
where L is a linear map. The tensor product is uniquely characterized by this property, up to a unique isomorphism.
This construction is related to repeated tensor products of two spaces. For example, if V1, V2, and V3 are three vector spaces, then there are (natural) isomorphisms
More generally, the tensor product of an arbitrary indexed family Vi, i ∈ I, is defined to be universal with respect to multilinear mappings of the direct product
A permutation
σ of the set {1, 2, ..., n} determines a mapping of the nth Cartesian power of V
defined by
Let
be the natural multilinear embedding of the Cartesian power of V into the tensor power of V. Then, by the universal property, there is a unique morphism
such that
The morphism τσ is called the braiding map associated to the permutation σ.
on V is an element of a vector space of the form
for non-negative integers r and s. There is a general formula for the components
of a (tensor) product of two (or more) tensor
s. For example, if F and G are two covariant tensors of rank m and n (respectively) (i.e. F ∈ Tm0, and G ∈ Tn0), then the components of their tensor product are given by
In this example, it is assumed that there is a chosen basis B of the vector space V, and the basis on any tensor space Tsr is tacitly assumed to be the standard one
(this basis is described in the article on Kronecker products).
Thus, the components of the tensor product of two tensors are the ordinary product of the components of each tensor.
Note that in the tensor product, the factor F consumes the first rank(F) indices, and the factor G consumes the next rank(G) indices, so
The tensor
may be naturally viewed as a module for the Lie algebra
by means of the diagonal action: for simplicity let us assume 
, then, for each 
,
where
is the transpose
of
, that is, in terms of the obvious pairing on 
,
.
There is a canonical isomorphism
given by
Under this isomorphism, every
may be first viewed as an endomorphism of 
and then viewed as an endomorphism of 
. In fact it is the adjoint representation 
of 
.
and
The tensor product inherits all the indices of its factors.
imposed upon it, in which each element of the first matrix is replaced by the second matrix, scaled by that element. For matrices
and 
this is:
For example, the tensor product of two two-dimensional square matrices:
The resultant rank is at most 4, and the resultant dimension 16. Here rank denotes the tensor rank (number of requisite indices), while the matrix rank counts the number of degrees of freedom in the resulting array.
A representative case is the Kronecker product
of any two rectangular arrays, considered as matrices. A dyadic product is the special case of the tensor product between two vectors of the same dimension.
and 
their tensor product is the multilinear function
(the dual space
of
, containing all linear functional
s on that space) is naturally identified with the space of all bilinear functionals on
. In other words, every bilinear functional is a functional on the tensor product, and vice versa.
Whenever
and 
are finite dimensional, there is a natural isomorphism
between
and 
, whereas for vector spaces of arbitrary dimension we only have an inclusion 
. So, the tensors of the linear functionals are bilinear functionals. This gives us a new way to look at the space of bilinear functionals, as a tensor product itself.
s of the tensor space, which are frequently useful. See wedge product for the first major example. Another would be the treatment of algebraic forms as symmetric tensors.
over a ring
R.
the tensor product is expressed as
(for example 
or 
). In J the tensor product is the dyadic form of */ (for example a */ b or a */ b */ c).
Note that J's treatment also allows the representation of some tensor fields, as a and b may be functions instead of constants. This product of two functions is a derived function, and if a and b are differentiable, then a*/b is differentiable.
However, these kinds of notation are not universally present in array languages. Other array languages may require explicit treatment of indices (for example, MATLAB
), and/or may not support higher-order functions such as the Jacobian derivative (for example, Fortran
/APL).
Mathematics
Mathematics is the study of quantity, space, structure, and change. Mathematicians seek out patterns and formulate new conjectures. Mathematicians resolve the truth or falsity of conjectures by mathematical proofs, which are arguments sufficient to convince other mathematicians of their validity...
, the tensor product, denoted by ⊗, may be applied in different contexts to vector
Vector space
A vector space is a mathematical structure formed by a collection of vectors: objects that may be added together and multiplied by numbers, called scalars in this context. Scalars are often taken to be real numbers, but one may also consider vector spaces with scalar multiplication by complex...
s, matrices
Matrix (mathematics)
In mathematics, a matrix is a rectangular array of numbers, symbols, or expressions. The individual items in a matrix are called its elements or entries. An example of a matrix with six elements isMatrices of the same size can be added or subtracted element by element...
, tensor
Tensor
Tensors are geometric objects that describe linear relations between vectors, scalars, and other tensors. Elementary examples include the dot product, the cross product, and linear maps. Vectors and scalars themselves are also tensors. A tensor can be represented as a multi-dimensional array of...
s, vector space
Vector space
A vector space is a mathematical structure formed by a collection of vectors: objects that may be added together and multiplied by numbers, called scalars in this context. Scalars are often taken to be real numbers, but one may also consider vector spaces with scalar multiplication by complex...
s, algebras
Algebra over a field
In mathematics, an algebra over a field is a vector space equipped with a bilinear vector product. That is to say, it isan algebraic structure consisting of a vector space together with an operation, usually called multiplication, that combines any two vectors to form a third vector; to qualify as...
, topological vector space
Topological vector space
In mathematics, a topological vector space is one of the basic structures investigated in functional analysis...
s, and module
Module (mathematics)
In abstract algebra, the concept of a module over a ring is a generalization of the notion of vector space, wherein the corresponding scalars are allowed to lie in an arbitrary ring...
s, among many other structures or objects. In each case the significance of the symbol is the same: the most general bilinear operation
Bilinear operator
In mathematics, a bilinear operator is a function combining elements of two vector spaces to yield an element of a third vector space that is linear in each of its arguments. Matrix multiplication is an example.-Definition:...
. In some contexts, this product is also referred to as outer product
Outer product
In linear algebra, the outer product typically refers to the tensor product of two vectors. The result of applying the outer product to a pair of vectors is a matrix...
. The term "tensor product" is also used in relation to monoidal categories
Monoidal category
In mathematics, a monoidal category is a category C equipped with a bifunctorwhich is associative, up to a natural isomorphism, and an object I which is both a left and right identity for ⊗, again up to a natural isomorphism...
.
Tensor product of vector spaces
The tensor product V ⊗ W of two vector spaceVector space
A vector space is a mathematical structure formed by a collection of vectors: objects that may be added together and multiplied by numbers, called scalars in this context. Scalars are often taken to be real numbers, but one may also consider vector spaces with scalar multiplication by complex...
s V and W over a field
Field (mathematics)
In abstract algebra, a field is a commutative ring whose nonzero elements form a group under multiplication. As such it is an algebraic structure with notions of addition, subtraction, multiplication, and division, satisfying certain axioms...
K can be defined by the method of generators and relations.
To construct V ⊗ W, one begins with the set of ordered pairs in the Cartesian product
Cartesian product
In mathematics, a Cartesian product is a construction to build a new set out of a number of given sets. Each member of the Cartesian product corresponds to the selection of one element each in every one of those sets...
V × W. For the purposes of this construction, regard this Cartesian product as a set rather than a vector space. The free vector space F on V × W is defined by taking the vector space in which the elements of V × W are a basis
Basis (linear algebra)
In linear algebra, a basis is a set of linearly independent vectors that, in a linear combination, can represent every vector in a given vector space or free module, or, more simply put, which define a "coordinate system"...
. In set-builder notation
Set-builder notation
In set theory and its applications to logic, mathematics, and computer science, set-builder notation is a mathematical notation for describing a set by stating the properties that its members must satisfy...
,
where we have used the symbol e(v,w) to emphasize that these are taken to be linearly independent
Linear independence
In linear algebra, a family of vectors is linearly independent if none of them can be written as a linear combination of finitely many other vectors in the collection. A family of vectors which is not linearly independent is called linearly dependent...
by definition for distinct (v, w) ∈ V × W.
The tensor product arises by defining the following four equivalence relations in F(V × W):
where v1, v2, and w1, w2 are vectors from V and W (respectively), and c is from the underlying field K. Denoting by R the space generated by these four equivalence relations, the tensor product of the two vector spaces V and W is then the quotient space
Quotient space
In topology and related areas of mathematics, a quotient space is, intuitively speaking, the result of identifying or "gluing together" certain points of a given space. The points to be identified are specified by an equivalence relation...
It is also called the tensor product space of V and W and is a vector space (which can be verified by directly checking the vector space axioms). The tensor product of two elements v and w is the equivalence class (e(v,w) + R) of e(v,w) in V ⊗ W, denoted v ⊗ w. This notation can somewhat obscure the fact that tensors are always coset
Coset
In mathematics, if G is a group, and H is a subgroup of G, and g is an element of G, thenA coset is a left or right coset of some subgroup in G...
s: manipulations performed via the representatives (v,w) must always be checked that they do not depend on the particular choice of representative.
The space R is mapped to zero in V ⊗ W, so that the above three equivalence relations become equalities in the tensor product space:
Given bases {vi} and {wi} for V and W respectively, the tensors {vi ⊗ wj} form a basis for V ⊗ W (generally ordered so that vi ⊗ wj+1 comes before vi+1 ⊗ wj). The dimension of the tensor product therefore is the product of dimensions of the original spaces; for instance Rm ⊗ Rn will have dimension mn.
Elements of V ⊗ W are sometimes referred to as tensors, although this term refers to many other related concepts as well. An element of V ⊗ W of the form v ⊗ w is called a pure or simple tensor. In general, an element of the tensor product space is not a pure tensor, but rather a finite linear combination of pure tensors. That is to say, if v1 and v2 are linearly independent, and w1 and w2 are also linearly independent, then v1 ⊗ w1 + v2 ⊗ w2 cannot be written as a pure tensor. The number of simple tensors required to express an element of a tensor product is called the tensor rank, (not to be confused with tensor order, which is the number of spaces one has taken the product of, in this case 2; in notation, the number of indices) and for linear operators or matrices, thought of as (1,1) tensors (elements of the space V ⊗ V*), it agrees with matrix rank.
Characterization by a universal propertyUniversal propertyIn various branches of mathematics, a useful construction is often viewed as the “most efficient solution” to a certain problem. The definition of a universal property uses the language of category theory to make this notion precise and to study it abstractly.This article gives a general treatment...
The tensor product of V and W can be defined (up to isomorphism) by any pair (L, φ), with L a vector space on K and a bilinear map such that
- for any K-vector space Z and any bilinear map
, there exists a unique linear map 
verifying 
In this sense, φ is the most general bilinear map that can be built from VxW.
It is easy to check that
satisfies this universal property.As a functorFunctorIn category theory, a branch of mathematics, a functor is a special type of mapping between categories. Functors can be thought of as homomorphisms between categories, or morphisms when in the category of small categories....
The tensor product also operates on linear maps between vector spaces. Specifically, given two linear maps S : V → X and T : W → Y between vector spaces, the tensor product of the two linear maps S and T is a linear mapdefined by
In this way, the tensor product becomes a bifunctor from the category of vector spaces to itself, covariant in both arguments.
The Kronecker product of two matrices is the matrix of the tensor product of the two corresponding linear maps under a standard choice of bases of the tensor products (see the article on Kronecker products).
More than two vector spaces
The construction and the universal property of the tensor product can be extended to allow for more than two vector spaces. For example, suppose that V1, V2, and V3 are three vector spaces. The tensor product V1 ⊗ V2 ⊗ V3 is defined along with a trilinear mapping from the direct productDirect product
In mathematics, one can often define a direct product of objectsalready known, giving a new one. This is generally the Cartesian product of the underlying sets, together with a suitably defined structure on the product set....
so that, any trilinear map F from the direct product to a vector space W
factors uniquely as
where L is a linear map. The tensor product is uniquely characterized by this property, up to a unique isomorphism.
This construction is related to repeated tensor products of two spaces. For example, if V1, V2, and V3 are three vector spaces, then there are (natural) isomorphisms
More generally, the tensor product of an arbitrary indexed family Vi, i ∈ I, is defined to be universal with respect to multilinear mappings of the direct product
Tensor powers and braiding
Let n be a non-negative integer. The nth tensor power of the vector space V is the n-fold tensor product of V with itself. That isA permutation
Permutation
In mathematics, the notion of permutation is used with several slightly different meanings, all related to the act of permuting objects or values. Informally, a permutation of a set of objects is an arrangement of those objects into a particular order...
σ of the set {1, 2, ..., n} determines a mapping of the nth Cartesian power of V
defined by
Let
be the natural multilinear embedding of the Cartesian power of V into the tensor power of V. Then, by the universal property, there is a unique morphism
such that
The morphism τσ is called the braiding map associated to the permutation σ.
Tensor product of two tensors
A tensorTensor
Tensors are geometric objects that describe linear relations between vectors, scalars, and other tensors. Elementary examples include the dot product, the cross product, and linear maps. Vectors and scalars themselves are also tensors. A tensor can be represented as a multi-dimensional array of...
on V is an element of a vector space of the form
for non-negative integers r and s. There is a general formula for the components
Coordinate vector
In linear algebra, a coordinate vector is an explicit representation of a vector in an abstract vector space as an ordered list of numbers or, equivalently, as an element of the coordinate space Fn....
of a (tensor) product of two (or more) tensor
Tensor
Tensors are geometric objects that describe linear relations between vectors, scalars, and other tensors. Elementary examples include the dot product, the cross product, and linear maps. Vectors and scalars themselves are also tensors. A tensor can be represented as a multi-dimensional array of...
s. For example, if F and G are two covariant tensors of rank m and n (respectively) (i.e. F ∈ Tm0, and G ∈ Tn0), then the components of their tensor product are given by
In this example, it is assumed that there is a chosen basis B of the vector space V, and the basis on any tensor space Tsr is tacitly assumed to be the standard one
(this basis is described in the article on Kronecker products).
Thus, the components of the tensor product of two tensors are the ordinary product of the components of each tensor.
Note that in the tensor product, the factor F consumes the first rank(F) indices, and the factor G consumes the next rank(G) indices, so
The tensor
may be naturally viewed as a module for the Lie algebraLie algebra
In mathematics, a Lie algebra is an algebraic structure whose main use is in studying geometric objects such as Lie groups and differentiable manifolds. Lie algebras were introduced to study the concept of infinitesimal transformations. The term "Lie algebra" was introduced by Hermann Weyl in the...
by means of the diagonal action: for simplicity let us assume , then, for each ,where
is the transposeTranspose
In linear algebra, the transpose of a matrix A is another matrix AT created by any one of the following equivalent actions:...
of
, that is, in terms of the obvious pairing on ,.There is a canonical isomorphism
given byUnder this isomorphism, every
may be first viewed as an endomorphism of and then viewed as an endomorphism of . In fact it is the adjoint representation of .Example
Let U be a tensor of type (1,1) with components Uαβ, and let V be a tensor of type (1,0) with components Vγ. Thenand
The tensor product inherits all the indices of its factors.
Kronecker product of two matrices
With matrices this operation is usually called the Kronecker product, a term used to make clear that the result has a particular block structureBlock structure
* In mathematics, block structure is a possible property of matrices - see block matrix.* In computer science, a programming language has block structure if it features blocks which can be nested to any depth....
imposed upon it, in which each element of the first matrix is replaced by the second matrix, scaled by that element. For matrices
and this is:For example, the tensor product of two two-dimensional square matrices:
The resultant rank is at most 4, and the resultant dimension 16. Here rank denotes the tensor rank (number of requisite indices), while the matrix rank counts the number of degrees of freedom in the resulting array.
A representative case is the Kronecker product
Kronecker product
In mathematics, the Kronecker product, denoted by ⊗, is an operation on two matrices of arbitrary size resulting in a block matrix. It gives the matrix of the tensor product with respect to a standard choice of basis. The Kronecker product should not be confused with the usual matrix...
of any two rectangular arrays, considered as matrices. A dyadic product is the special case of the tensor product between two vectors of the same dimension.
Tensor product of multilinear maps
Given multilinear maps and their tensor product is the multilinear functionRelation with the dual space
In the discussion on the universal property, replacing X by the underlying scalar field of V and W yields that the space (the dual spaceDual space
In mathematics, any vector space, V, has a corresponding dual vector space consisting of all linear functionals on V. Dual vector spaces defined on finite-dimensional vector spaces can be used for defining tensors which are studied in tensor algebra...
of
, containing all linear functionalFunctional (mathematics)
In mathematics, and particularly in functional analysis, a functional is a map from a vector space into its underlying scalar field. In other words, it is a function that takes a vector as its input argument, and returns a scalar...
s on that space) is naturally identified with the space of all bilinear functionals on
. In other words, every bilinear functional is a functional on the tensor product, and vice versa.Whenever
and are finite dimensional, there is a natural isomorphismIsomorphism
In abstract algebra, an isomorphism is a mapping between objects that shows a relationship between two properties or operations. If there exists an isomorphism between two structures, the two structures are said to be isomorphic. In a certain sense, isomorphic structures are...
between
and , whereas for vector spaces of arbitrary dimension we only have an inclusion . So, the tensors of the linear functionals are bilinear functionals. This gives us a new way to look at the space of bilinear functionals, as a tensor product itself.Types of tensors
Linear subspaces of the bilinear operators (or in general, multilinear operators) determine natural quotient spaceQuotient space
In topology and related areas of mathematics, a quotient space is, intuitively speaking, the result of identifying or "gluing together" certain points of a given space. The points to be identified are specified by an equivalence relation...
s of the tensor space, which are frequently useful. See wedge product for the first major example. Another would be the treatment of algebraic forms as symmetric tensors.
Over more general rings
The notation ⊗R refers to a tensor product of modulesTensor product of modules
In mathematics, the tensor product of modules is a construction that allows arguments about bilinear maps to be carried out in terms of linear maps...
over a ring
Ring (mathematics)
In mathematics, a ring is an algebraic structure consisting of a set together with two binary operations usually called addition and multiplication, where the set is an abelian group under addition and a semigroup under multiplication such that multiplication distributes over addition...
R.
Array programming languages
Array programming languages may have this pattern built in. For example, in APLAPL programming language
APL is an interactive array-oriented language and integrated development environment, which is available from a number of commercial and noncommercial vendors and for most computer platforms. It is based on a mathematical notation developed by Kenneth E...
the tensor product is expressed as
(for example or ). In J the tensor product is the dyadic form of */ (for example a */ b or a */ b */ c).Note that J's treatment also allows the representation of some tensor fields, as a and b may be functions instead of constants. This product of two functions is a derived function, and if a and b are differentiable, then a*/b is differentiable.
However, these kinds of notation are not universally present in array languages. Other array languages may require explicit treatment of indices (for example, MATLAB
MATLAB
MATLAB is a numerical computing environment and fourth-generation programming language. Developed by MathWorks, MATLAB allows matrix manipulations, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs written in other languages,...
), and/or may not support higher-order functions such as the Jacobian derivative (for example, Fortran
Fortran
Fortran is a general-purpose, procedural, imperative programming language that is especially suited to numeric computation and scientific computing...
/APL).
See also
- Extension of scalarsExtension of scalarsIn abstract algebra, extension of scalars is a means of producing a module over a ring S from a module over another ring R, given a homomorphism f : R \to S between them...
- Tensor product of R-algebras
- Tensor product of fieldsTensor product of fieldsIn abstract algebra, the theory of fields lacks a direct product: the direct product of two fields, considered as a ring is never itself a field. On the other hand it is often required to 'join' two fields K and L, either in cases where K and L are given as subfields of a larger field M, or when K...
- Tensor product of graphsTensor product of graphsIn graph theory, the tensor product G × H of graphs G and H is a graph such that* the vertex set of G × H is the Cartesian product V × V; and...
- Tensor product of Hilbert spacesTensor product of Hilbert spacesIn mathematics, and in particular functional analysis, the tensor product of Hilbert spaces is a way to extend the tensor product construction so that the result of taking a tensor product of two Hilbert space is another Hilbert space. Roughly speaking, the tensor product is the completion of the...
- Tensor product of line bundles
- Tensor product of modulesTensor product of modulesIn mathematics, the tensor product of modules is a construction that allows arguments about bilinear maps to be carried out in terms of linear maps...
- Tensor product of quadratic formsTensor product of quadratic formsThe tensor product of quadratic forms is most easily understood when one views the quadratic forms as quadratic spaces. So, if and are quadratic spaces, which V,W vector spaces, then the tensor product is a quadratic form q on the tensor product of vector spaces V \otimes W.It is defined in such...
- Topological tensor productTopological tensor productIn mathematics, there are usually many different ways to construct a topological tensor product of two topological vector spaces. For Hilbert spaces or nuclear spaces there is a simple well-behaved theory of tensor products , but for general Banach spaces or locally convex topological vector space...
- Dyadic product
- Tensor contractionTensor contractionIn multilinear algebra, a tensor contraction is an operation on one or more tensors that arises from the natural pairing of a finite-dimensional vector space and its dual. In components, it is expressed as a sum of products of scalar components of the tensor caused by applying the summation...