Winner-take-all
Encyclopedia
Winner-take-all is a computational principle applied in computational models of neural network
s by which neuron
s in a layer compete with each others for activation. In the classical form, only the neuron with the highest activation stays active while all other neurons shut down, however other variations that allow more than one neuron to be active do exist, for example the soft winner take-all, by which a power function is applied to the neurons.
s winner-take-all networks are a case of competitive learning
in recurrent neural network
s. Output nodes in the network mutually inhibit each other, while simultaneously activating themselves through reflexive connections. After some time, only one node in the output layer will be active, namely the one corresponding to the strongest input. Thus the network uses nonlinear inhibition to pick out the largest of a set of inputs. Winner-take-all is a general computational primitive that can be implemented using different types of neural network models, including both continuous-time and spiking networks (Grossberg, 1973; Oster et al. 2009).
Winner-take-all networks are commonly used in computational models of the brain, particularly for distributed decision-making in the cortex
. Important examples include hierarchical models of vision (Riesenhuber et al. 1999), and models of selective attention and recognition (Carpenter and Grossberg, 1987; Itti et al. 1998). They are also common in artificial neural networks and neuromorphic analog VLSI circuits. It has been formally proven that the winner-take-all operation is computationally powerful compared to other nonlinear operations, such as thresholding (Maass 2000).
In many practical cases, there is not only a single neuron which becomes the only active one but there are exactly k neurons which become active for a fixed number k. This principle is referred to as k-winners-take-all.
winner-take-all circuit is shown on the right. This circuit was originally proposed by Lazzaro et al. (1989) using MOS transistors biased to operate in the weak-inversion or subthreshold regime. In the particular case shown there are only two inputs (IIN,1 and IIN,2), but the circuit can be easily extended to multiple inputs in a straightforward way. It operates on continuous-time input signals (currents) in parallel, using only two transistors per input. In addition, the bias current IBIAS is set by a single global transistor that is common to all the inputs.
The largest of the input currents sets the common potential VC. As a result, the corresponding output carries almost all the bias current, while the other outputs have currents that are close to zero. Thus, the circuit selects the larger of the two input currents, i.e., if IIN,1 > IIN,2, we get IOUT,1 = IBIAS and IOUT,2 = 0. Similarly, if IIN,2 > IIN,1, we get IOUT,1 = 0 and IOUT,2 = IBIAS.
A SPICE
-based DC simulation of the CMOS winner-take-all circuit in the two-input case is shown on the right. As shown in the top subplot, the input IIN,1 was fixed at 6nA, while IIN,2 was linearly increased from 0 to 10nA. The bottom subplot shows the two output currents. As expected, the output corresponding to the larger of the two inputs carries the entire bias current (10nA in this case), forcing the other output current nearly to zero.
It is axiomatic that in the electronic commerce market, early dominant players such as AOL
or Yahoo!
get most of the rewards. By 1998, one study found the top 5% of all web sites garnered more than 74% of all traffic.
The winner take all hypothesis suggests that once a technology or a firm gets ahead, it will do better and better over time, whereas lagging technology and firms will fall further behind.
Neural network
The term neural network was traditionally used to refer to a network or circuit of biological neurons. The modern usage of the term often refers to artificial neural networks, which are composed of artificial neurons or nodes...
s by which neuron
Neuron
A neuron is an electrically excitable cell that processes and transmits information by electrical and chemical signaling. Chemical signaling occurs via synapses, specialized connections with other cells. Neurons connect to each other to form networks. Neurons are the core components of the nervous...
s in a layer compete with each others for activation. In the classical form, only the neuron with the highest activation stays active while all other neurons shut down, however other variations that allow more than one neuron to be active do exist, for example the soft winner take-all, by which a power function is applied to the neurons.
Neural Networks
In the theory of artificial neural networkArtificial neural network
An artificial neural network , usually called neural network , is a mathematical model or computational model that is inspired by the structure and/or functional aspects of biological neural networks. A neural network consists of an interconnected group of artificial neurons, and it processes...
s winner-take-all networks are a case of competitive learning
Competitive learning
Competitive learning is a form of unsupervised learning in artificial neural networks, in which nodes compete for the right to respond to a subset of the input data. A variant of Hebbian learning, competitive learning works by increasing the specialization of each node in the network...
in recurrent neural network
Recurrent neural network
A recurrent neural network is a class of neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behavior. Unlike feedforward neural networks, RNNs can use their internal memory to process...
s. Output nodes in the network mutually inhibit each other, while simultaneously activating themselves through reflexive connections. After some time, only one node in the output layer will be active, namely the one corresponding to the strongest input. Thus the network uses nonlinear inhibition to pick out the largest of a set of inputs. Winner-take-all is a general computational primitive that can be implemented using different types of neural network models, including both continuous-time and spiking networks (Grossberg, 1973; Oster et al. 2009).
Winner-take-all networks are commonly used in computational models of the brain, particularly for distributed decision-making in the cortex
Cortex (anatomy)
In anatomy and zoology the cortex is the outermost layer of an organ. Organs with well-defined cortical layers include kidneys, adrenal glands, ovaries, the thymus, and portions of the brain, including the cerebral cortex, the most well-known of all cortices.The cerebellar cortex is the thin gray...
. Important examples include hierarchical models of vision (Riesenhuber et al. 1999), and models of selective attention and recognition (Carpenter and Grossberg, 1987; Itti et al. 1998). They are also common in artificial neural networks and neuromorphic analog VLSI circuits. It has been formally proven that the winner-take-all operation is computationally powerful compared to other nonlinear operations, such as thresholding (Maass 2000).
In many practical cases, there is not only a single neuron which becomes the only active one but there are exactly k neurons which become active for a fixed number k. This principle is referred to as k-winners-take-all.
Circuit example
A simple, but popular CMOSCMOS
Complementary metal–oxide–semiconductor is a technology for constructing integrated circuits. CMOS technology is used in microprocessors, microcontrollers, static RAM, and other digital logic circuits...
winner-take-all circuit is shown on the right. This circuit was originally proposed by Lazzaro et al. (1989) using MOS transistors biased to operate in the weak-inversion or subthreshold regime. In the particular case shown there are only two inputs (IIN,1 and IIN,2), but the circuit can be easily extended to multiple inputs in a straightforward way. It operates on continuous-time input signals (currents) in parallel, using only two transistors per input. In addition, the bias current IBIAS is set by a single global transistor that is common to all the inputs.
The largest of the input currents sets the common potential VC. As a result, the corresponding output carries almost all the bias current, while the other outputs have currents that are close to zero. Thus, the circuit selects the larger of the two input currents, i.e., if IIN,1 > IIN,2, we get IOUT,1 = IBIAS and IOUT,2 = 0. Similarly, if IIN,2 > IIN,1, we get IOUT,1 = 0 and IOUT,2 = IBIAS.
A SPICE
SPICE
SPICE is a general-purpose, open source analog electronic circuit simulator.It is a powerful program that is used in integrated circuit and board-level design to check the integrity of circuit designs and to predict circuit behavior.- Introduction :Unlike board-level designs composed of discrete...
-based DC simulation of the CMOS winner-take-all circuit in the two-input case is shown on the right. As shown in the top subplot, the input IIN,1 was fixed at 6nA, while IIN,2 was linearly increased from 0 to 10nA. The bottom subplot shows the two output currents. As expected, the output corresponding to the larger of the two inputs carries the entire bias current (10nA in this case), forcing the other output current nearly to zero.
Other uses
In stereo matching algorithms, following the taxonomy proposed by Scharstein et al. (IJCV 2002), winner-take-all is a local method for disparity computation. Adopting a winner-take-all strategy, the disparity associated with the minimum or maximum cost value is selected at each pixel.It is axiomatic that in the electronic commerce market, early dominant players such as AOL
AOL
AOL Inc. is an American global Internet services and media company. AOL is headquartered at 770 Broadway in New York. Founded in 1983 as Control Video Corporation, it has franchised its services to companies in several nations around the world or set up international versions of its services...
or Yahoo!
Yahoo!
Yahoo! Inc. is an American multinational internet corporation headquartered in Sunnyvale, California, United States. The company is perhaps best known for its web portal, search engine , Yahoo! Directory, Yahoo! Mail, Yahoo! News, Yahoo! Groups, Yahoo! Answers, advertising, online mapping ,...
get most of the rewards. By 1998, one study found the top 5% of all web sites garnered more than 74% of all traffic.
The winner take all hypothesis suggests that once a technology or a firm gets ahead, it will do better and better over time, whereas lagging technology and firms will fall further behind.