Statistical distance
Encyclopedia
In statistics
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....

, probability theory
Probability theory
Probability theory is the branch of mathematics concerned with analysis of random phenomena. The central objects of probability theory are random variables, stochastic processes, and events: mathematical abstractions of non-deterministic events or measured quantities that may either be single...

, and information theory
Information theory
Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental limits on signal processing operations such as compressing data and on reliably storing and...

, a statistical distance quantifies the distance between two statistical objects, which can be two samples
Sample (statistics)
In statistics, a sample is a subset of a population. Typically, the population is very large, making a census or a complete enumeration of all the values in the population impractical or impossible. The sample represents a subset of manageable size...

, two random variable
Random variable
In probability and statistics, a random variable or stochastic variable is, roughly speaking, a variable whose value results from a measurement on some type of random process. Formally, it is a function from a probability space, typically to the real numbers, which is measurable functionmeasurable...

s, or two probability distribution
Probability distribution
In probability theory, a probability mass, probability density, or probability distribution is a function that describes the probability of a random variable taking certain values....

s, for example.

Metrics

A metric on a set X is a function
Function (mathematics)
In mathematics, a function associates one quantity, the argument of the function, also known as the input, with another quantity, the value of the function, also known as the output. A function assigns exactly one output to each input. The argument and the value may be real numbers, but they can...

 (called the distance function or simply distance)

d : X × XR

(where R is the set of real number
Real number
In mathematics, a real number is a value that represents a quantity along a continuum, such as -5 , 4/3 , 8.6 , √2 and π...

s). For all x, y, z in X, this function is required to satisfy the following conditions:
  1. d(x, y) ≥ 0     (non-negativity)
  2. d(x, y) = 0   if and only if   x = y     (identity of indiscernibles
    Identity of indiscernibles
    The identity of indiscernibles is an ontological principle which states that two or more objects or entities are identical if they have all their properties in common. That is, entities x and y are identical if any predicate possessed by x is also possessed by y and vice versa...

    . Note that condition 1 and 2 together produce positive definiteness)
  3. d(x, y) = d(y, x)     (symmetry
    Symmetric relation
    In mathematics, a binary relation R over a set X is symmetric if it holds for all a and b in X that if a is related to b then b is related to a.In mathematical notation, this is:...

    )
  4. d(x, z) ≤ d(x, y) + d(y, z)     (subadditivity / triangle inequality
    Triangle inequality
    In mathematics, the triangle inequality states that for any triangle, the sum of the lengths of any two sides must be greater than or equal to the length of the remaining side ....

    ).

Distances: Generalized metrics

Many statistical distances are not metric
Metric (mathematics)
In mathematics, a metric or distance function is a function which defines a distance between elements of a set. A set with a metric is called a metric space. A metric induces a topology on a set but not all topologies can be generated by a metric...

s, because they lack one or more properties of proper metrics. For example, pseudometric
Pseudometric space
In mathematics, a pseudometric space is a generalized metric space in which the distance between two distinct points can be zero. In the same way as every normed space is a metric space, every seminormed space is a pseudometric space...

s can violate the "positive definiteness" (alternatively, "identity of indescernibles" property); quasimetrics can violate the symmetry property; and semimetrics can violate the triangle inequality. Some statistical distances are referred to as divergence
Divergence (statistics)
In statistics and information geometry, divergence or a contrast function is a function which establishes the “distance” of one probability distribution to the other on a statistical manifold...

s
.

Examples

Some important statistical distances include the following:
  • f-divergence
    F-divergence
    In probability theory, an ƒ-divergence is a function Df that measures the difference between two probability distributions P and Q...

    : includes
    • Kullback–Leibler divergence
      Kullback–Leibler divergence
      In probability theory and information theory, the Kullback–Leibler divergence is a non-symmetric measure of the difference between two probability distributions P and Q...

    • Hellinger distance
      Hellinger distance
      In probability and statistics, the Hellinger distance is used to quantify the similarity between two probability distributions. It is a type of f-divergence...

    • Total variation distance
  • Rényi's divergence
    Rényi entropy
    In information theory, the Rényi entropy, a generalisation of Shannon entropy, is one of a family of functionals for quantifying the diversity, uncertainty or randomness of a system...

  • Jensen–Shannon divergence
    Jensen–Shannon divergence
    In probability theory and statistics, the Jensen–Shannon divergence is a popular method of measuring the similarity between two probability distributions. It is also known as information radius or total divergence to the average. It is based on the Kullback–Leibler divergence, with the notable ...

  • Lévy–Prokhorov metric
  • Bhattacharyya distance
    Bhattacharyya distance
    In statistics, the Bhattacharyya distance measures the similarity of two discrete or continuous probability distributions. It is closely related to the Bhattacharyya coefficient which is a measure of the amount of overlap between two statistical samples or populations. Both measures are named after A...

  • Wasserstein metric
    Wasserstein metric
    In mathematics, the Wasserstein metric is a distance function defined between probability distributions on a given metric space M....

    : also known as the Kantorovich metric, or earth mover's distance
    Earth Mover's Distance
    In computer science, the earth mover's distance is a measure of the distance between two probability distributions over a region D. In mathematics, this is known as the Wasserstein metric...

  • Energy distance
    Energy distance
    Energy distance is a statistical distance between probability distributions. If X and Y are independent random vectors in Rd, with cumulative distribution functions F and G respectively, then the energy distance between the distributions F and G is definedwhere X, X' are independent and identically...



Other approaches
  • Signal-to-noise ratio
    Signal-to-noise ratio
    Signal-to-noise ratio is a measure used in science and engineering that compares the level of a desired signal to the level of background noise. It is defined as the ratio of signal power to the noise power. A ratio higher than 1:1 indicates more signal than noise...

     distance
  • Mahalanobis distance
    Mahalanobis distance
    In statistics, Mahalanobis distance is a distance measure introduced by P. C. Mahalanobis in 1936. It is based on correlations between variables by which different patterns can be identified and analyzed. It gauges similarity of an unknown sample set to a known one. It differs from Euclidean...

  • Distance correlation
    Distance correlation
    In statistics and in probability theory, distance correlation is a measure of statistical dependence between two random variables or two random vectors of arbitrary, not necessarily equal dimension. Its important property is that this measure of dependence is zero if and only if the random...

     is a measure of dependence between two random variables, it is zero if and only if the random variables are independent.
  • The continuous ranked probability score is a measure how good forecasts that are expressed as probability distributions are in matching observed outcomes. Both the location and spread of the forecast distribution are taken into account in judging how close the distribution is the observed value: see probabilistic forecasting
    Probabilistic forecasting
    Probabilistic forecasting summarises what is known, or opinions about, future events. In contrast to a single-valued forecasts , probabilistic forecasts assign a probability to each of a number of different outcomes,...

    .
  • Lukaszyk–Karmowski metric is a function defining a distance between two random variable
    Random variable
    In probability and statistics, a random variable or stochastic variable is, roughly speaking, a variable whose value results from a measurement on some type of random process. Formally, it is a function from a probability space, typically to the real numbers, which is measurable functionmeasurable...

    s or two random vectors. It does not satisfy the identity of indiscernibles
    Identity of indiscernibles
    The identity of indiscernibles is an ontological principle which states that two or more objects or entities are identical if they have all their properties in common. That is, entities x and y are identical if any predicate possessed by x is also possessed by y and vice versa...

     condition of the metric and is zero if and only if both its arguments are certain events described by Dirac delta density probability distribution function
    Probability distribution function
    Depending upon which text is consulted, a probability distribution function is any of:* a probability distribution function,* a cumulative distribution function,* a probability mass function, or* a probability density function....

    s.
The source of this article is wikipedia, the free encyclopedia.  The text of this article is licensed under the GFDL.
 
x
OK