Sensory substitution
Encyclopedia
Sensory substitution means to transform the characteristics of one sensory modality into stimuli of another sensory modality. It is hoped that sensory substitution systems can help handicapped people by restoring their ability to perceive a certain defective sensory modality by using sensory information from a functioning sensory modality. A sensory substitution system consists of three parts: a sensor, a coupling system, and a stimulator. The sensor records stimuli and gives them to a coupling system which interprets these signals and transmits them to a stimulator. In case the sensor obtains signals of a kind not originally available to the bearer it is a case of sensory augmentation. Sensory substitution concerns human perception
and the plasticity
of the human brain; and therefore, allows us to study these aspects of neuroscience more through neuroimaging
.
as a means of using one sensory modality, mainly tactition, to gain environmental information to be used by another sensory modality, mainly vision
. The first sensory substitution system was developed by Bach-y-Rita et al. as a means of brain plasticity in congenitally blind individuals. After this historic invention, sensory substitution has been the basis of many studies investigating perceptive and cognitive neuroscience
. Since then, sensory substitution has contributed to the study of brain function, human cognition
and rehabilitation.
for visions and cochlea
for hearing) to brain. Since the vision processing pathways are still intact, a person who has lost the ability to retrieve data from the retina can still see subjective images by using data gathered from other sensory modalities such as touch or audition.
In a regular visual system, the data collected by the retina is converted into an electrical stimulus in the optic nerve
and relayed to the brain, which re-creates the image and perceives it. Because it is the brain that is responsible for the final perception, sensory substitution is possible. During sensory substitution an intact sensory modality relays information to the visual perception areas of the brain so that the person can perceive to see. With sensory substitution, information gained from one sensory modality can reach brain structures physiologically related to other sensory modalities. Touch-to-visual sensory substitution transfers information from touch receptors to the visual cortex for interpretation and perception. For example, through fMRI, we can determine which parts of the brain are activated during sensory perception. In blind persons, we can see that while they are only receiving tactile information, their visual cortex is also activated as they perceive to see objects. We can also have touch to touch sensory substitution where information from touch receptors of one region can be used to perceive touch in another region. For example, in one experiment by Bach-y-Rita, he was able to restore the touch perception in a patient who lost peripheral sensation from leprosy.
istic presentations, game
s, and augmented reality
. Some examples are substitution of visual stimuli to audio or tactile, and of audio stimuli to tactile. Some of the most popular are probably Paul Bach-y-Rita's Tactile Vision Sensory Substitution (TVSS), developed with Carter Collins at Smith-Kettlewell Institute
and Peter Meijer's Seeing with Sound approach (The vOICe). Technical developments, such as miniaturization
and electrical stimulation help the advance of sensory substitution devices.
In sensory substitution systems, we generally have sensors that collect the data from the external environment. This data is then relayed to a coupling system that interprets and transduces the information and then replays it to a stimulator. This stimulator ultimately stimulates a functioning sensory modality. After training, people learn to use the information gained from this stimulation to experience a perception of the sensation they lack instead of the actually stimulated sensation. For example, a leprosy patient, whose perception of peripheral touch was restored, was equipped with a glove containing artificial contact sensors coupled to skin sensory receptors on the forehead (which was stimulated). After training and acclimation, the patient was able to experience data from the glove as if it was originating in the fingertips while ignoring the sensations in the forehead.
, Meissner's corpuscle
, Ruffini ending
s, Merkel nerve ending
s, free nerve ending
s, and tactile disks. These receptors are mainly characterized by their ability to adapt to stimuli and their thresholds. Because of the relative high thresholds of most these receptors and their rapid adaptation to stimulus, the human body requires rapidly changing tactile stimulation systems.
There have been two different types of stimulators: electrotactile or vibrotactile. Electrotactile stimulators use direct electrical stimulation of the nerve ending in the skin to initiate the action potentials; the sensation triggered, burn, itch, pain, pressure etc. depends on the stimulating voltage. Vibrotactile stimulators use pressure and the properties of the mechanoreceptors of the skin to initiate action potentials. There are advantages and disadvantages for both these stimulation systems. With the electrotactile stimulating systems a lot of factors effect the sensation triggered: stimulating voltage, current, waveform, electrode size, material, contact force, skin location, thickness and hydration. Electrotactile stimulation may involve the direct stimulation of the nerves (percutaneous
), or through the skin (transcutaneous). Percutaneous application causes additional distress to the patient, and is a major disadvantage of this approach. Furthermore, stimulation of the skin without insertion leads to the need for high voltage stimulation because of the high impedance of the dry skin, unless the tongue is used as a receptor, which requires only about 3% as much voltage. This latter technique is undergoing clinical trials for various applications. Alternatively, the roof of the mouth has been proposed as another area where low currents can be felt.
Electrostatic arrays are explored as human-computer interaction devices for touch screens. These are based on a phenomenon called electrovibration, which allows microamperre-level currents to be felt as roughness on a surface.
Vibrotactile systems use the properties of mechanoreceptors in the skin so they have fewer parameters that need to be monitored as compared to electrotactile stimulation. However, vibrotactile stimulation systems need to account for the rapid adaptation of the tactile sense.
Another important aspect of tactile sensory substitution systems is the location of the tactile stimulation. Tactile receptors are abundant on the fingertips, face, and tongue while sparse on the back, legs and arms. It is essential to take into account the spatial resolution of the receptor as it has a major effect on the resolution of the sensory substitution.
Below you can find some descriptions of current tactile substitution systems.
of his blind subject. Recently, several new systems have been developed that interface the tactile image to tactile receptors on different areas of the body such as the on the chest, brow, fingertip, abdomen, and forehead. The tactile image is produced by hundreds of activators placed on the person. The activators are solenoid
s of one millimeter diameter. In experiments, blind
(or blindfold
ed) subjects equipped with the TVSS can learn to detect shapes and to orient themselves. In the case of simple geometric shapes, it took around 50 trials to achieve 100 percent correct recognition. To identify objects in different orientations requires several hours of learning.
A system using the tongue as the human-machine interface is most practical. The tongue-machine interface is both protected by the closed mouth and the saliva in the mouth provides a good electrolytic environment that ensures good electrode contact. Results from a study by Bach-y-Rita et al. show that electrotactile stimulation of the tongue required 3% of the voltage required to stimulate the finger. Also, since it is more practical to wear an orthodontic retainer holding the stimulation system than an apparatus strapped to other parts of the body, the tongue-machine interface is more popular among TVSS systems.
This tongue TVSS system works by delivering electrotactile stimuli to the dorsum of the tongue via a flexible electrode array
placed in the mouth. This electrode array is connected to a Tongue Display Unit [TDU] via a ribbon cable passing out of the mouth. A video camera records a picture, transfers it to the TDU for conversion into a tactile image. The tactile image is then projected onto the tongue via the ribbon cable where the tongue’s receptors pick up the signal. After training, subjects are able to associate certain types of stimuli to certain types of visual images. In this way, tactile sensation can be used for visual perception.
Sensory substitutions have also been successful with the emergence of wearable haptic actuators like vibrotactile motors, solenoids, peltier diodes, etc. At the Center for Cognitive Ubiquitous Computing @ Arizona State University
researchers have developed technologies that enable people who are blind to perceive social situational information using wearable vibrotactile belts (Haptic Belt) and gloves (VibroGlove). Both technologies use miniature cameras that are mounted on a pair of glasses worn by the user who is blind. The Haptic Belt provides vibrations that convey the direction and distance at which a person is standing in front of a user, while the VibroGlove uses spatio-temporal mapping of vibration patterns to convey facial expressions of the interaction partner.
s or adverse reactions to antibiotics suffer from bilateral vestibular damage (BVD). They experience difficulty maintaining posture, unstable gait, and oscillopsia
. Tyler et al. studied the restitution of postural control through a tactile for vestibular sensory substitution. Because BVD patients cannot integrate visual and tactile cues, they have a lot of difficulty standing. Using a head-mounted accelerometer
and a brain-machine interface that employs electrotactile stimulation on the tongue, information about head-body orientation was relayed to the patient so that a new source of data is available to orient themselves and maintain good posture.
Other applications of sensory substitution systems can be seen in function robotic prostheses for patients with high level quadriplegia. These robotic arms have several mechanisms of slip detection, vibration and texture detection that they relay to the patient through feedback. After more research and development, the information from these arms can be used by patients to perceive that they are holding and manipulating objects while their robotic arm actually accomplishes the task.
The vOICe vision technology is one of several approaches towards sensory substitution (vision substitution) for the blind that aims to provide synthetic vision
to the user by means of a non-invasive visual prosthesis. The vOICe converts live camera views from a video camera into soundscapes. This system uses general video to audio mapping by associating height to pitch and brightness with loudness in a left-to-right scan of any video frame. Views are typically refreshed about once per second with a typical image resolution of up to 60 x 60 pixels as can be proven by spectrographic analysis.
Neuroscience and psychology research indicate recruitment of relevant brain areas in seeing with sound, as well as functional improvement through training. The ultimate goal is to provide synthetic vision with truly visual sensations by exploiting the neural plasticity of the human brain. Neuroscience
research has shown that the visual cortex
of even adult blind people can become responsive to sound, and “seeing with sound” might reinforce this in a visual sense with live video from a head-mounted camera encoded in sound. The extent to which cortical plasticity indeed allows for functionally relevant rewiring or remapping of the human brain is still largely unknown and is being investigated in an open collaboration with research partners around the world.
One suggestion for increasing the relative efficiency of the resulting visual stimuli is to adjust the visual field by using an accelerometer to provide a steady image even if the head is moved.
Connecting an infrared sensor to adjust the camera position to match eye movements would also help here; this idea was suggested by A.De Guerin on 30/10/2011
Another successful visual-to-auditory sensory substitution device is the Prosthesis Substituting Vision for Audition (PSVA). This system utilizes a head-mounted TV camera that allows real-time, online translation of visual patterns into sound. While the patient moves around, the device captures visual frames at a high frequency and generates the corresponding complex sounds that allow recognition. Visual stimuli are transduced into auditory stimuli with the use of a system that uses pixel to frequency relationship and couples a rough model of the human retina with an inverse model of the cochlea.
The sound produced by this software is a mixture of sinusoidal sounds produced by virtual "sources", corresponding each to a "receptive field" in the image. Each receptive field is a set of localized pixels. The sound's amplitude is determined by the mean luminosity of the pixels of the corresponding receptive field. The frequency and the inter-aural disparity are determined by the center of gravity of the co-ordinates of the receptive field's pixels in the image (see "There is something out there: distal attribution in sensory substitution, twenty years later"; Auvray M., Hanneton S., Lenay C., O'Regan K. Journal of Integrative Neuroscience 4 (2005) 505-21). The Vibe is an Open Source project hosted by Sourceforge.
Other approaches to the substitution of hearing for vision use binaural directional cues, much as natural human echolocation
does. An example of the latter approach is the "SeeHear" chip from Caltech.
Other visual-auditory substitution devices deviate from the vOICe's greyscale mapping of images. Zach Capalbo's Kromophone uses a basic color spectrum correlating to different sounds and timbres to give users perceptual information beyond the vOICe's capabilities.
, that signals can be employed from force/touch indicators on a robot hand as a means of communication.
Active work in this direction is being conducted by, among others, the e-sense project of the Open University
and Edinburgh University, and the feelSpace project of the University of Osnabrück
.
The findings of research into sensory augmentation (as well as sensory substitution in general) that investigate the emergence of perceptual experience (qualia) from the activity of neurons have implications for the understanding of consciousness.
Significant performance improvements in navigational tests were observed (over and above those experienced by control subjects during the same period with the same training) and, for half of the participants, the perception of the belt's vibration underwent a profound change from simple tactile innervation to approach a genuine and direct sense of allocentric orientation: in other words, could perceive north as an entity distinct from the vibrating transducer on the waist, like one perceives a glass on a table as an entity distinct from the impact of reflected photons on the retina. Further, tests of the influence of the belt information on the rotational nystagmus effect suggested that, after training, the processing of the belt information became subcognitive.
Perception
Perception is the process of attaining awareness or understanding of the environment by organizing and interpreting sensory information. All perception involves signals in the nervous system, which in turn result from physical stimulation of the sense organs...
and the plasticity
Neuroplasticity
Neuroplasticity is a non-specific neuroscience term referring to the ability of the brain and nervous system in all species to change structurally and functionally as a result of input from the environment. Plasticity occurs on a variety of levels, ranging from cellular changes involved in...
of the human brain; and therefore, allows us to study these aspects of neuroscience more through neuroimaging
Neuroimaging
Neuroimaging includes the use of various techniques to either directly or indirectly image the structure, function/pharmacology of the brain...
.
History
Sensory Substitution was introduced in the '60s by Paul Bach-y-RitaPaul Bach-y-Rita
Paul Bach-y-Rita was an American neuroscientist whose most notable work was in the field of neuroplasticity. Bach-y-Rita was one of the first to seriously study the idea of neuroplasticity , and to introduce sensory substitution as a tool to treat patients suffering from neurological...
as a means of using one sensory modality, mainly tactition, to gain environmental information to be used by another sensory modality, mainly vision
Visual perception
Visual perception is the ability to interpret information and surroundings from the effects of visible light reaching the eye. The resulting perception is also known as eyesight, sight, or vision...
. The first sensory substitution system was developed by Bach-y-Rita et al. as a means of brain plasticity in congenitally blind individuals. After this historic invention, sensory substitution has been the basis of many studies investigating perceptive and cognitive neuroscience
Cognitive neuroscience
Cognitive neuroscience is an academic field concerned with the scientific study of biological substrates underlying cognition, with a specific focus on the neural substrates of mental processes. It addresses the questions of how psychological/cognitive functions are produced by the brain...
. Since then, sensory substitution has contributed to the study of brain function, human cognition
Cognition
In science, cognition refers to mental processes. These processes include attention, remembering, producing and understanding language, solving problems, and making decisions. Cognition is studied in various disciplines such as psychology, philosophy, linguistics, and computer science...
and rehabilitation.
Physiology of sensory substitution
When a person becomes blind or deaf they generally do not lose the ability to hear or see, they simply lose their ability to transmit the sensory signals from the periphery (retinaRetina
The vertebrate retina is a light-sensitive tissue lining the inner surface of the eye. The optics of the eye create an image of the visual world on the retina, which serves much the same function as the film in a camera. Light striking the retina initiates a cascade of chemical and electrical...
for visions and cochlea
Cochlea
The cochlea is the auditory portion of the inner ear. It is a spiral-shaped cavity in the bony labyrinth, making 2.5 turns around its axis, the modiolus....
for hearing) to brain. Since the vision processing pathways are still intact, a person who has lost the ability to retrieve data from the retina can still see subjective images by using data gathered from other sensory modalities such as touch or audition.
In a regular visual system, the data collected by the retina is converted into an electrical stimulus in the optic nerve
Optic nerve
The optic nerve, also called cranial nerve 2, transmits visual information from the retina to the brain. Derived from the embryonic retinal ganglion cell, a diverticulum located in the diencephalon, the optic nerve doesn't regenerate after transection.-Anatomy:The optic nerve is the second of...
and relayed to the brain, which re-creates the image and perceives it. Because it is the brain that is responsible for the final perception, sensory substitution is possible. During sensory substitution an intact sensory modality relays information to the visual perception areas of the brain so that the person can perceive to see. With sensory substitution, information gained from one sensory modality can reach brain structures physiologically related to other sensory modalities. Touch-to-visual sensory substitution transfers information from touch receptors to the visual cortex for interpretation and perception. For example, through fMRI, we can determine which parts of the brain are activated during sensory perception. In blind persons, we can see that while they are only receiving tactile information, their visual cortex is also activated as they perceive to see objects. We can also have touch to touch sensory substitution where information from touch receptors of one region can be used to perceive touch in another region. For example, in one experiment by Bach-y-Rita, he was able to restore the touch perception in a patient who lost peripheral sensation from leprosy.
Technological support
In order to have sensory substitution and stimulate the brain without intact sensory organs to relay the information, it is also possible to develop machines that do the signal transduction. This brain–machine interface is where external signals are collected and transduced into electrical signals for the brain to interpret. Generally a camera or a microphone is used to collect visual or auditory stimuli that are used to replace lost sensory information. The visual or auditory data collected from the sensors is transduced into tactile stimuli that are then relayed to the brain for visual and auditory perception. This type of sensory substitution is only possible due to the plasticity of the brain.Brain plasticity
Brain plasticity is the brain’s ability to adapt to the complete absence or the deterioration of a sense. Sensory substitution is therefore most likely explained through the study of brain plasticity. Cortical re-mapping or reorganization takes place when the brain experiences some sort of deterioration. This is an evolutionary mechanism that allows people with the deprivation of a sense to adapt and compensate by using other senses. Functional imaging of congenitally blind patients showed a cross-modal recruitment of the occipital cortex during the realization perceptual tasks such as Braille reading, tactile perception, tactual object recognition, sound localization, and sound discrimination. This shows that blind people can use their occipital lobe, generally used for vision, to perceive objects though the use of other sensory modalities, which would explain their oft-displayed propensity towards increased strength of the other senses.Perception versus sensing
While talking about the physiological aspects of sensory substitution, it is essential to distinguish between sensing and perceiving. The general question posed by this differentiation is: Are blind people seeing or perceiving to see by putting together different sensory data? While sensation comes in one modality – visual, auditory, tactile etc. – perception due to sensory substitution is not one modality but a result of cross-modal interactions. Therefore, we can say that while sensory substitution for vision induces visual-like perception in sighted individual, it induces auditory or tactile perception in blind individuals. In short, blind people perceive to see through touch and audition with sensory substitution.Different applications of sensory substitution
Applications are not restricted to handicapped persons, but also include artArt
Art is the product or process of deliberately arranging items in a way that influences and affects one or more of the senses, emotions, and intellect....
istic presentations, game
Game
A game is structured playing, usually undertaken for enjoyment and sometimes used as an educational tool. Games are distinct from work, which is usually carried out for remuneration, and from art, which is more often an expression of aesthetic or ideological elements...
s, and augmented reality
Augmented reality
Augmented reality is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is...
. Some examples are substitution of visual stimuli to audio or tactile, and of audio stimuli to tactile. Some of the most popular are probably Paul Bach-y-Rita's Tactile Vision Sensory Substitution (TVSS), developed with Carter Collins at Smith-Kettlewell Institute
Smith-Kettlewell Institute
The Smith-Kettlewell Eye Research Institute in San Francisco was formed in 1959 by Arthur Jampolsky on the former campus of the Stanford Medical School.The current interim director of Smith-Kettlewell is Arthur Jampolsky.Major Work:...
and Peter Meijer's Seeing with Sound approach (The vOICe). Technical developments, such as miniaturization
Miniaturization
Miniaturization is the creation of ever-smaller scales for mechanical, optical, and electronic products and devices...
and electrical stimulation help the advance of sensory substitution devices.
In sensory substitution systems, we generally have sensors that collect the data from the external environment. This data is then relayed to a coupling system that interprets and transduces the information and then replays it to a stimulator. This stimulator ultimately stimulates a functioning sensory modality. After training, people learn to use the information gained from this stimulation to experience a perception of the sensation they lack instead of the actually stimulated sensation. For example, a leprosy patient, whose perception of peripheral touch was restored, was equipped with a glove containing artificial contact sensors coupled to skin sensory receptors on the forehead (which was stimulated). After training and acclimation, the patient was able to experience data from the glove as if it was originating in the fingertips while ignoring the sensations in the forehead.
Tactile sensory substitution systems
To understand tactile sensory substitution it is essential to understand some basic physiology of the tactile receptors of the skin. There are six basic types of tactile receptors: Pacinian corpusclePacinian corpuscle
Lamellar corpuscles or Pacinian corpuscles are one of the four major types of mechanoreceptor. They are nerve endings in the skin, responsible for sensitivity to vibration and pressure. Vibrational role may be used to detect surface, e.g., rough vs...
, Meissner's corpuscle
Meissner's corpuscle
Meissner's corpuscles are a type of mechanoreceptor. They are a type of nerve ending in the skin that is responsible for sensitivity to light touch. In particular, they have highest sensitivity when sensing vibrations lower than 50 Hertz...
, Ruffini ending
Ruffini ending
The Bulbous corpuscle or Ruffini ending or Ruffini corpuscle is a class of slowly adapting mechanoreceptor thought to exist only in the glabrous dermis and subcutaneous tissue of humans...
s, Merkel nerve ending
Merkel nerve ending
Merkel nerve endings are mechanoreceptors found in the skin and mucosa of vertebrates that provide touch information to the brain. The information they provide are those regarding pressure and texture. Each ending consists of a Merkel cell in close apposition with an enlarged nerve terminal...
s, free nerve ending
Free nerve ending
A free nerve ending is an unspecialized, afferent nerve ending, meaning it brings information from the body's periphery toward the brain. They function as cutaneous receptors and are essentially used by vertebrates to detect pain.-Structure:...
s, and tactile disks. These receptors are mainly characterized by their ability to adapt to stimuli and their thresholds. Because of the relative high thresholds of most these receptors and their rapid adaptation to stimulus, the human body requires rapidly changing tactile stimulation systems.
There have been two different types of stimulators: electrotactile or vibrotactile. Electrotactile stimulators use direct electrical stimulation of the nerve ending in the skin to initiate the action potentials; the sensation triggered, burn, itch, pain, pressure etc. depends on the stimulating voltage. Vibrotactile stimulators use pressure and the properties of the mechanoreceptors of the skin to initiate action potentials. There are advantages and disadvantages for both these stimulation systems. With the electrotactile stimulating systems a lot of factors effect the sensation triggered: stimulating voltage, current, waveform, electrode size, material, contact force, skin location, thickness and hydration. Electrotactile stimulation may involve the direct stimulation of the nerves (percutaneous
Percutaneous
In surgery, percutaneous pertains to any medical procedure where access to inner organs or other tissue is done via needle-puncture of the skin, rather than by using an "open" approach where inner organs or tissue are exposed .The percutaneous approach is commonly used in vascular procedures...
), or through the skin (transcutaneous). Percutaneous application causes additional distress to the patient, and is a major disadvantage of this approach. Furthermore, stimulation of the skin without insertion leads to the need for high voltage stimulation because of the high impedance of the dry skin, unless the tongue is used as a receptor, which requires only about 3% as much voltage. This latter technique is undergoing clinical trials for various applications. Alternatively, the roof of the mouth has been proposed as another area where low currents can be felt.
Electrostatic arrays are explored as human-computer interaction devices for touch screens. These are based on a phenomenon called electrovibration, which allows microamperre-level currents to be felt as roughness on a surface.
Vibrotactile systems use the properties of mechanoreceptors in the skin so they have fewer parameters that need to be monitored as compared to electrotactile stimulation. However, vibrotactile stimulation systems need to account for the rapid adaptation of the tactile sense.
Another important aspect of tactile sensory substitution systems is the location of the tactile stimulation. Tactile receptors are abundant on the fingertips, face, and tongue while sparse on the back, legs and arms. It is essential to take into account the spatial resolution of the receptor as it has a major effect on the resolution of the sensory substitution.
Below you can find some descriptions of current tactile substitution systems.
Tactile–visual substitution
One of the earliest and most well known form of sensory substitution devices was Paul Bach-y-Rita’s TVSS that converted the image from a video camera into a tactile image and coupled it to the tactile receptors on the backBack
- People :* Adam Back, British cryptographer* Charles Back, South African winemaker* Chris Back , Australian politician* Ernst Emil Alexander Back , German physicist* Frédéric Back , Canadian animator...
of his blind subject. Recently, several new systems have been developed that interface the tactile image to tactile receptors on different areas of the body such as the on the chest, brow, fingertip, abdomen, and forehead. The tactile image is produced by hundreds of activators placed on the person. The activators are solenoid
Solenoid
A solenoid is a coil wound into a tightly packed helix. In physics, the term solenoid refers to a long, thin loop of wire, often wrapped around a metallic core, which produces a magnetic field when an electric current is passed through it. Solenoids are important because they can create...
s of one millimeter diameter. In experiments, blind
Blindness
Blindness is the condition of lacking visual perception due to physiological or neurological factors.Various scales have been developed to describe the extent of vision loss and define blindness...
(or blindfold
Blindfold
A blindfold is a garment, usually of cloth, tied to one's head to cover the eyes to disable the wearer's sight. It can be worn when the eyes are in a closed state and thus prevents the wearer from opening them...
ed) subjects equipped with the TVSS can learn to detect shapes and to orient themselves. In the case of simple geometric shapes, it took around 50 trials to achieve 100 percent correct recognition. To identify objects in different orientations requires several hours of learning.
A system using the tongue as the human-machine interface is most practical. The tongue-machine interface is both protected by the closed mouth and the saliva in the mouth provides a good electrolytic environment that ensures good electrode contact. Results from a study by Bach-y-Rita et al. show that electrotactile stimulation of the tongue required 3% of the voltage required to stimulate the finger. Also, since it is more practical to wear an orthodontic retainer holding the stimulation system than an apparatus strapped to other parts of the body, the tongue-machine interface is more popular among TVSS systems.
This tongue TVSS system works by delivering electrotactile stimuli to the dorsum of the tongue via a flexible electrode array
Electrode array
An electrode array is a configuration of electrodes used for measuring either an electric current or voltage. Some electrode arrays can operate in a bidirectional fashion, in that they can also be used to provide a stimulating pattern of electric current or voltage.Common arrays...
placed in the mouth. This electrode array is connected to a Tongue Display Unit [TDU] via a ribbon cable passing out of the mouth. A video camera records a picture, transfers it to the TDU for conversion into a tactile image. The tactile image is then projected onto the tongue via the ribbon cable where the tongue’s receptors pick up the signal. After training, subjects are able to associate certain types of stimuli to certain types of visual images. In this way, tactile sensation can be used for visual perception.
Sensory substitutions have also been successful with the emergence of wearable haptic actuators like vibrotactile motors, solenoids, peltier diodes, etc. At the Center for Cognitive Ubiquitous Computing @ Arizona State University
Arizona State University
Arizona State University is a public research university located in the Phoenix Metropolitan Area of the State of Arizona...
researchers have developed technologies that enable people who are blind to perceive social situational information using wearable vibrotactile belts (Haptic Belt) and gloves (VibroGlove). Both technologies use miniature cameras that are mounted on a pair of glasses worn by the user who is blind. The Haptic Belt provides vibrations that convey the direction and distance at which a person is standing in front of a user, while the VibroGlove uses spatio-temporal mapping of vibration patterns to convey facial expressions of the interaction partner.
Tactile–auditory substitution
While there are no tactile-auditory substitution system currently available, recent experiments by Schurmann et al. show that tactile senses can activate the human auditory cortex. Currently vibrotactile stimuli can be used to facilitate hearing in normal and hearing-impaired people. To test for the auditory areas activated by touch, Schurmann et al. tested subjects while stimulating their fingers and palms with vibration bursts and their finger tips with tactile pressure. They found that tactile stimulation of the fingers lead to activation of the auditory belt area, which suggests that there is a relationship between audition and tactition. Therefore, future research can be done to investigate the likelihood of a tactile-auditory sensory substitution system. One promising invention is the 'Sense organs synthesizer', which may be viewed at http://freepatentsonline.com/y20020173823 and select Download PDF 20020173823 for descriptions with diagrams. Full normal hearing range of nine octaves is delivered via 216 electrodes to sequential touch nerve zones, next to the spine. Inventor is incorporating a nonprofit organization in August 2010, to build prototypes for 'proof of concept' considerations.Tactile–vestibular substitution
Some people with balance disorderBalance disorder
A balance disorder is a disturbance that causes an individual to feel unsteady, for example when standing or walking. It may be accompanied by feelings of giddiness or wooziness, or having a sensation of movement, spinning, or floating...
s or adverse reactions to antibiotics suffer from bilateral vestibular damage (BVD). They experience difficulty maintaining posture, unstable gait, and oscillopsia
Oscillopsia
Oscillopsia is a visual disturbance in which objects in the visual field appear to oscillate. The severity of the effect may range from a mild blurring to rapid and periodic jumping...
. Tyler et al. studied the restitution of postural control through a tactile for vestibular sensory substitution. Because BVD patients cannot integrate visual and tactile cues, they have a lot of difficulty standing. Using a head-mounted accelerometer
Accelerometer
An accelerometer is a device that measures proper acceleration, also called the four-acceleration. This is not necessarily the same as the coordinate acceleration , but is rather the type of acceleration associated with the phenomenon of weight experienced by a test mass that resides in the frame...
and a brain-machine interface that employs electrotactile stimulation on the tongue, information about head-body orientation was relayed to the patient so that a new source of data is available to orient themselves and maintain good posture.
Tactile–tactile substitution to restore peripheral sensation
Touch to touch sensory substitution is where information from touch receptors of one region can be used to perceive touch in another. For example, in one experiment by Bach-y-Rita, the touch perception was restored in a patient who lost peripheral sensation from leprosy. For example, this leprosy patient was equipped with a glove containing artificial contact sensors coupled to skin sensory receptors on the forehead (which was stimulated). After training and acclimation, the patient was able to experience data from the glove as if it was originating in the fingertips while ignoring the sensations in the forehead. After two days of training one of the leprosy subjects reported “the wonderful sensation of touching his wife, which he had been unable to experience for 20 years.”Tactile feedback system for prosthetic limbs
The development of new technologies has now made it plausible to provide patients with prosthetic arms with tactile and kinesthetic sensibilities. While this is not purely a sensory substitution system, it uses the same principles to restore perception of senses. Some tactile feedback methods of restoring a perception of touch to amputees would be direct or micro stimulation of the tactile nerve afferents.Other applications of sensory substitution systems can be seen in function robotic prostheses for patients with high level quadriplegia. These robotic arms have several mechanisms of slip detection, vibration and texture detection that they relay to the patient through feedback. After more research and development, the information from these arms can be used by patients to perceive that they are holding and manipulating objects while their robotic arm actually accomplishes the task.
Auditory sensory substitution systems
Auditory sensory substitution systems like the tactile sensory substitution systems aim to use one sensory modality to compensate for the lack of another sensory modality in order to gain a perception of one that is lacking. With auditory sensory substitution, we use visual or tactile sensors to detect and store information about the external environment. This information is then transduced by brain-machine interfaces into auditory signals that are then relayed via the auditory receptors to the brain.Auditory vision substitution
Auditory vision substitution aims to use the sense of hearing to convey visual information to the blind.The vOICe
The vOICe vision technology is one of several approaches towards sensory substitution (vision substitution) for the blind that aims to provide synthetic vision
Visual perception
Visual perception is the ability to interpret information and surroundings from the effects of visible light reaching the eye. The resulting perception is also known as eyesight, sight, or vision...
to the user by means of a non-invasive visual prosthesis. The vOICe converts live camera views from a video camera into soundscapes. This system uses general video to audio mapping by associating height to pitch and brightness with loudness in a left-to-right scan of any video frame. Views are typically refreshed about once per second with a typical image resolution of up to 60 x 60 pixels as can be proven by spectrographic analysis.
Neuroscience and psychology research indicate recruitment of relevant brain areas in seeing with sound, as well as functional improvement through training. The ultimate goal is to provide synthetic vision with truly visual sensations by exploiting the neural plasticity of the human brain. Neuroscience
Neuroscience
Neuroscience is the scientific study of the nervous system. Traditionally, neuroscience has been seen as a branch of biology. However, it is currently an interdisciplinary science that collaborates with other fields such as chemistry, computer science, engineering, linguistics, mathematics,...
research has shown that the visual cortex
Visual cortex
The visual cortex of the brain is the part of the cerebral cortex responsible for processing visual information. It is located in the occipital lobe, in the back of the brain....
of even adult blind people can become responsive to sound, and “seeing with sound” might reinforce this in a visual sense with live video from a head-mounted camera encoded in sound. The extent to which cortical plasticity indeed allows for functionally relevant rewiring or remapping of the human brain is still largely unknown and is being investigated in an open collaboration with research partners around the world.
One suggestion for increasing the relative efficiency of the resulting visual stimuli is to adjust the visual field by using an accelerometer to provide a steady image even if the head is moved.
Connecting an infrared sensor to adjust the camera position to match eye movements would also help here; this idea was suggested by A.De Guerin on 30/10/2011
PSVA
Another successful visual-to-auditory sensory substitution device is the Prosthesis Substituting Vision for Audition (PSVA). This system utilizes a head-mounted TV camera that allows real-time, online translation of visual patterns into sound. While the patient moves around, the device captures visual frames at a high frequency and generates the corresponding complex sounds that allow recognition. Visual stimuli are transduced into auditory stimuli with the use of a system that uses pixel to frequency relationship and couples a rough model of the human retina with an inverse model of the cochlea.
The Vibe
The sound produced by this software is a mixture of sinusoidal sounds produced by virtual "sources", corresponding each to a "receptive field" in the image. Each receptive field is a set of localized pixels. The sound's amplitude is determined by the mean luminosity of the pixels of the corresponding receptive field. The frequency and the inter-aural disparity are determined by the center of gravity of the co-ordinates of the receptive field's pixels in the image (see "There is something out there: distal attribution in sensory substitution, twenty years later"; Auvray M., Hanneton S., Lenay C., O'Regan K. Journal of Integrative Neuroscience 4 (2005) 505-21). The Vibe is an Open Source project hosted by Sourceforge.
Other systems
Other approaches to the substitution of hearing for vision use binaural directional cues, much as natural human echolocation
Human echolocation
Human echolocation is the ability of humans to detect objects in their environment by sensing echoes from those objects. By actively creating sounds – for example, by tapping their canes, lightly stomping their foot or making clicking noises with their mouths – people trained to orientate with...
does. An example of the latter approach is the "SeeHear" chip from Caltech.
Other visual-auditory substitution devices deviate from the vOICe's greyscale mapping of images. Zach Capalbo's Kromophone uses a basic color spectrum correlating to different sounds and timbres to give users perceptual information beyond the vOICe's capabilities.
Nervous system implants
By means of stimulating electrodes implanted into the human nervous system, it is possible to apply current pulses to be learned and reliably recognized by the recipient. It has been shown successfully in experimentation, by Kevin WarwickKevin Warwick
Kevin Warwick is a British scientist and professor of cybernetics at the University of Reading, Reading, Berkshire, United Kingdom...
, that signals can be employed from force/touch indicators on a robot hand as a means of communication.
Criticism
It has been argued that the term "substitution" is misleading, as it is merely an "addition" or "supplementation" not a substitution of a sensory modality.Sensory Augmentation
Building upon the research conducted on sensory substition, investigations into the possibility of augmenting the body's sensory apparatus are now beginning. The intention is to extend the body's ability to sense aspects of the environment that are not normally perceivable by the body in its natural state.Active work in this direction is being conducted by, among others, the e-sense project of the Open University
Open University
The Open University is a distance learning and research university founded by Royal Charter in the United Kingdom...
and Edinburgh University, and the feelSpace project of the University of Osnabrück
University of Osnabrück
The University of Osnabrück is a public university located in the city of Osnabrück in Lower Saxony, Germany.In 2010 it was attended by 9,298 students. In 2009, the staff of 1,570 consisted of 214 professors, 662 additional academic personnel and 694 non-academic personnel...
.
The findings of research into sensory augmentation (as well as sensory substitution in general) that investigate the emergence of perceptual experience (qualia) from the activity of neurons have implications for the understanding of consciousness.
Magnetic Perception
In 2005, the feelSpace group conducted a study of sensory augmentation with a vibrotactile magnetic compass belt worn around the waist. In this study, the participants were provided with the direction of magnetic north as a vibration on their waist.Significant performance improvements in navigational tests were observed (over and above those experienced by control subjects during the same period with the same training) and, for half of the participants, the perception of the belt's vibration underwent a profound change from simple tactile innervation to approach a genuine and direct sense of allocentric orientation: in other words, could perceive north as an entity distinct from the vibrating transducer on the waist, like one perceives a glass on a table as an entity distinct from the impact of reflected photons on the retina. Further, tests of the influence of the belt information on the rotational nystagmus effect suggested that, after training, the processing of the belt information became subcognitive.
See also
- Biological neural networkBiological neural networkIn neuroscience, a biological neural network describes a population of physically interconnected neurons or a group of disparate neurons whose inputs or signalling targets define a recognizable circuit. Communication between neurons often involves an electrochemical process...
- Brain implantBrain implantBrain implants, often referred to as neural implants, are technological devices that connect directly to a biological subject's brain - usually placed on the surface of the brain, or attached to the brain's cortex...
- Human echolocationHuman echolocationHuman echolocation is the ability of humans to detect objects in their environment by sensing echoes from those objects. By actively creating sounds – for example, by tapping their canes, lightly stomping their foot or making clicking noises with their mouths – people trained to orientate with...
, blind people navigating by listening to the echo of sounds
External links
- Tongue display for sensory substitution
- The vOICe auditory display for sensory substitution.
- Webpage of Laurent Renier
- Artificial Retinas
- Sensory Substitution:limits and perspectives C. Lenay et al.
- The Vibe
- feelSpace - The Magnetic Perception Group of the University of Osnabrück
- Kromophone at Gordon College's Philosophical Psychology Lab