Minimum Fisher Information
Encyclopedia
In information theory
, the principle of minimum Fisher information (MFI) is a variational principle
which, when applied with the proper constraints needed to reproduce empirically known expectation values, determines the best probability distribution
that characterizes the system. (See also Fisher information
.)
. They measure either the amount of positive information or of "missing" information an observer possesses with regards to any system of interest. The most famous IM is the so-called Shannon-entropy (1948), which determines how much additional information the observer still requires in order to have all the available knowledge regarding a given system S, when all he/she has is a probability density function
(PD) defined on appropriate elements of such system. This is then a "missing" information measure. The IM is a functional of the PD only. If the observer does not have such
a PD, but only a finite set of empirically determined mean values of the system, then a fundamental scientific principle called the Maximum Entropy
one (MaxEnt) asserts that the "best" PD is the one that, reproducing the known expectation values, maximizes otherwise Shannon's IM.
(FIM), named after Ronald Fisher
, (1925) is another kind of measure, in two respects, namely,
1) it reflects the amount of (positive) information of the observer,
2) it depends no only on the PD but also on its first derivatives, a property that makes it a local quantity (Shannon's is instead a global one).
The corresponding counterpart of MaxEnt is now the FIM-minimization, since Fisher's measure grows when Shannon's diminishes, and viceversa. The minimization here referred to (MFI) is an important theoretical tool in a manifold of disciplines, beginning with physics
. In a sense it is clearly superior to MaxEnt because the later procedure yields always as the solution an exponential PD, while the MFI solution is a differential equation
for the PD, which allows for greater flexibility and versatility.
(both equilibrium
and non-equilibrium
) can be derived from the MFI approach . Here FIM is specialized to the particular but important case of translation families, i.e., distribution functions whose form does not change under translational transformations. In this case, Fisher measure becomes shift-invariant. Such minimizing of Fisher's measure leads to a Schrödinger-like equation
for the probability amplitude, where the ground state describes equilibrium physics and the excited states account for non-equilibrium situations .
is introduced in the measure, leading for the first time an explanation of this regularity from first principles . It has been also show that MFI can be used to formulate a thermodynamics based on scale invariance instead of translational invariance, allowing the definition of the Scale-Free Ideal Gas
, the scale invariant equivalent of the Ideal Gas
.
Information theory
Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental limits on signal processing operations such as compressing data and on reliably storing and...
, the principle of minimum Fisher information (MFI) is a variational principle
Variational principle
A variational principle is a scientific principle used within the calculus of variations, which develops general methods for finding functions which minimize or maximize the value of quantities that depend upon those functions...
which, when applied with the proper constraints needed to reproduce empirically known expectation values, determines the best probability distribution
Probability distribution
In probability theory, a probability mass, probability density, or probability distribution is a function that describes the probability of a random variable taking certain values....
that characterizes the system. (See also Fisher information
Fisher information
In mathematical statistics and information theory, the Fisher information is the variance of the score. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior...
.)
Measures of information
Information measures (IM) are the most important tools of information theoryInformation theory
Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental limits on signal processing operations such as compressing data and on reliably storing and...
. They measure either the amount of positive information or of "missing" information an observer possesses with regards to any system of interest. The most famous IM is the so-called Shannon-entropy (1948), which determines how much additional information the observer still requires in order to have all the available knowledge regarding a given system S, when all he/she has is a probability density function
Probability density function
In probability theory, a probability density function , or density of a continuous random variable is a function that describes the relative likelihood for this random variable to occur at a given point. The probability for the random variable to fall within a particular region is given by the...
(PD) defined on appropriate elements of such system. This is then a "missing" information measure. The IM is a functional of the PD only. If the observer does not have such
a PD, but only a finite set of empirically determined mean values of the system, then a fundamental scientific principle called the Maximum Entropy
Principle of maximum entropy
In Bayesian probability, the principle of maximum entropy is a postulate which states that, subject to known constraints , the probability distribution which best represents the current state of knowledge is the one with largest entropy.Let some testable information about a probability distribution...
one (MaxEnt) asserts that the "best" PD is the one that, reproducing the known expectation values, maximizes otherwise Shannon's IM.
Fisher's information measure
Fisher's informationFisher information
In mathematical statistics and information theory, the Fisher information is the variance of the score. In Bayesian statistics, the asymptotic distribution of the posterior mode depends on the Fisher information and not on the prior...
(FIM), named after Ronald Fisher
Ronald Fisher
Sir Ronald Aylmer Fisher FRS was an English statistician, evolutionary biologist, eugenicist and geneticist. Among other things, Fisher is well known for his contributions to statistics by creating Fisher's exact test and Fisher's equation...
, (1925) is another kind of measure, in two respects, namely,
1) it reflects the amount of (positive) information of the observer,
2) it depends no only on the PD but also on its first derivatives, a property that makes it a local quantity (Shannon's is instead a global one).
The corresponding counterpart of MaxEnt is now the FIM-minimization, since Fisher's measure grows when Shannon's diminishes, and viceversa. The minimization here referred to (MFI) is an important theoretical tool in a manifold of disciplines, beginning with physics
Physics
Physics is a natural science that involves the study of matter and its motion through spacetime, along with related concepts such as energy and force. More broadly, it is the general analysis of nature, conducted in order to understand how the universe behaves.Physics is one of the oldest academic...
. In a sense it is clearly superior to MaxEnt because the later procedure yields always as the solution an exponential PD, while the MFI solution is a differential equation
Differential equation
A differential equation is a mathematical equation for an unknown function of one or several variables that relates the values of the function itself and its derivatives of various orders...
for the PD, which allows for greater flexibility and versatility.
Thermodynamics
Much effort has been devoted to Fisher's information measure, shedding much light upon the manifold physical applications . As a small sample, it can be shown that the whole field of thermodynamicsThermodynamics
Thermodynamics is a physical science that studies the effects on material bodies, and on radiation in regions of space, of transfer of heat and of work done on or by the bodies or radiation...
(both equilibrium
Thermodynamic equilibrium
In thermodynamics, a thermodynamic system is said to be in thermodynamic equilibrium when it is in thermal equilibrium, mechanical equilibrium, radiative equilibrium, and chemical equilibrium. The word equilibrium means a state of balance...
and non-equilibrium
Non-equilibrium thermodynamics
Non-equilibrium thermodynamics is a branch of thermodynamics that deals with systems that are not in thermodynamic equilibrium. Most systems found in nature are not in thermodynamic equilibrium; for they are changing or can be triggered to change over time, and are continuously and discontinuously...
) can be derived from the MFI approach . Here FIM is specialized to the particular but important case of translation families, i.e., distribution functions whose form does not change under translational transformations. In this case, Fisher measure becomes shift-invariant. Such minimizing of Fisher's measure leads to a Schrödinger-like equation
Schrödinger equation
The Schrödinger equation was formulated in 1926 by Austrian physicist Erwin Schrödinger. Used in physics , it is an equation that describes how the quantum state of a physical system changes in time....
for the probability amplitude, where the ground state describes equilibrium physics and the excited states account for non-equilibrium situations .
Scale-invariant phenomena
More recently, Zipf's law has been shown to arise as the variational solution of the MFI when scale invarianceScale invariance
In physics and mathematics, scale invariance is a feature of objects or laws that do not change if scales of length, energy, or other variables, are multiplied by a common factor...
is introduced in the measure, leading for the first time an explanation of this regularity from first principles . It has been also show that MFI can be used to formulate a thermodynamics based on scale invariance instead of translational invariance, allowing the definition of the Scale-Free Ideal Gas
Scale-Free Ideal Gas
The scale-free ideal gas is a physical model assuming a collection of non-interacting elements with an stochastic proportional growth. It is the scale-invariant version of an ideal gas...
, the scale invariant equivalent of the Ideal Gas
Ideal gas
An ideal gas is a theoretical gas composed of a set of randomly-moving, non-interacting point particles. The ideal gas concept is useful because it obeys the ideal gas law, a simplified equation of state, and is amenable to analysis under statistical mechanics.At normal conditions such as...
.