Cognitive Info-Communications (CogInfoCom)
Encyclopedia
Cognitive infocommunications (CogInfoCom) investigates the link between the research areas of infocommunications and cognitive science
s, as well as the various engineering applications which have emerged as the synergic combination of these sciences.
The primary goal of CogInfoCom is to provide a systematic view of how cognitive processes can co-evolve with infocommunications devices so that the capabilities of the human brain
may not only be extended through these devices, irrespective of geographical distance, but may also interact with the capabilities of any artificially cognitive system. This merging and extension of cognitive capabilities is targeted towards engineering applications in which artificial and/or natural cognitive systems are enabled to work together more effectively.
We define two important dimensions of cognitive infocommunications: the mode of communication and the type of communication.
The mode of communication refers to the actors at the two endpoints of communication:
The type of communication refers to the type of information that is conveyed between the two communicating entities, and the way in which it is carried out:
Remarks
The first draft definition of CogInfoCom was given in [1]. The above definition was finalized based on the paper with the joint participation of the Startup Committee and the participants of the 1st International Workshop on Cognitive Infocommunications, held in Tokyo
, Japan
in 2010.
The idea that the information systems we use need to be accommodated with artificially cognitive functionalities has culminated in the creation of cognitive informatics [5], [6].
With the strong support of cognitive science, results in cognitive informatics are contributing to the creation of more and more sophisticated artificially cognitive engineering systems. Given this trend, it is rapidly becoming clear that the amount of cognitive content handled by our engineering systems is reaching a point where the communication forms necessary to enable interaction with this content are becoming more and more complex [7], [8], [9], [10], [11].
The inspiration to create engineering systems capable of communicating with users in natural ways is not new. It is one of the primary goals of affective computing to endow information systems with emotion
, and to enable them to communicate these emotions in ways that resonate with the human users [12], [13]. Augmented cognition deals with modeling human attention and using results to create engineering systems which are capable of adapting based on human cognitive skills and momentary attention [14]. In addition, there are a host of research fields which concentrate less on modeling (human) psychological emotion, but aim to allow users to have a more tractable and pleasurable interaction with machines (e.g., human computer interaction, human robot interaction, interactive systems engineering) [15], [16]. Further fields specialize in the communication of hidden parameters to remote environments (e.g., sensory substitution and sensorimotor extension in engineering applications, iSpace research, multimodal interaction) [17], [18], [19], [20], [21], [22], [23].
In recent years, several applications have appeared in the technical literature which combine various aspects of the previously mentioned fields, but also extend them in significant
ways (e.g., [24], [25], [26], [27], [28], [29], [30], [23], [31], [32], [18], [33]). However, in these works, the fact that a new research field is emerging is only implicitly mentioned.
, media
, and communications were very different areas, treated by researchers from significantly different backgrounds. As a synthesis between pairs of these 3 disciplines, the fields of infocommunications, media informatics and media communications emerged in the latter half of the 20th century. The past evolution of these disciplines points towards their convergence in the near future, given that modern network services aim to provide a more holistic user experience, which presupposes achievements from these different fields [34], [35]. In place of these research areas, with the enormous growth in scope and generality of cognitive sciences in the past few decades, the new fields of cognitive media [36], [37], cognitive informatics and cognitive communications [38],[39] are gradually emerging. In a way analogous to the evolution of infocommunications, media informatics and media communications, we are seeing more and more examples of research achievements which can be categorized as results in cognitive infocommunications, cognitive media informatics and cognitive media communications, even if – as of yet – these fields are not always clearly defined [40], [41], [24],
[25], [29], [30], [42], [18], [43], [36].
The primary goal of CogInfoCom is to use information theoretical methods to synthesize research results in some of these areas, while aiming primarily to make use of these synthesized results in the design of engineering systems. It is novel in the sense that it views both the medium used for communication and the media which is communicated as entities which are interpreted by a cognitive system.
his / her natural cognitive system to communicate with the machine. In contrast, when 3D virtualization comes into play, the human is no
longer in direct connection with the machine; instead, he / she must communicate with its virtual representation. Thus, the problem of
human-machine interaction is transformed into a problem of human-virtual machine interaction. In such scenarios, the natural communication capabilities of the human become limited due to the restricted interface provided by the virtual representation of the machine (for instance, while the senses of vision and audition still receive considerable amount of information, the tactile and olfactory senses are almost completely restricted in virtual environments, i.e. it is usually not possible to touch or smell the virtual representation of a machine). For this reason, it becomes necessary to develop a virtual cognitive system which can extend the natural one so as to allow the human to effectively communicate with the virtual representation, which essentially becomes an infocommunication system in this extended scenario. One of the primary goals of cognitive infocommunications is to address this problem so that sensory modalities other than the ones normally used in virtual reality research can be employed for the perception of a large variety of information on the virtualized system.
[5], [6]. The main purpose of CI is to investigate the internal information storing and processing mechanisms in natural intelligent systems such as the human
brain.Much like CogInfoCom, CI also aims primarily to create numerically tractable models which are well grounded from an information theoretical point of view, and are amenable to engineering systems. The key difference between CI and CogInfoCom is that while the results of CI largely converge towards and support the creation of artificial cognitive systems, the goal of CogInfoCom is to enable these systems to communicate with each other and their users efficiently.
Thus, CogInfoCom builds on a large part of results in CI, since it deals with the communication space between the human cognitive system and other natural or artificial cognitive systems.
or some other telecommunication system, and a large variety of information types (e.g. metalinguistic information and background noises through sound, gesture-based metacommunication through video, etc.) are communicated to both ends of the line. In more futuristic applications, information from other sensory modalities (e.g. smells through the use of electronic nose
s and scent generators, tastes using equipment not yet available today) may also be communicated. Because the communicating actors are both human, the communication mode is intra-cognitive, and because the communicated information is shared using the same cognitive subsystems (i.e., the same sensory modalities) on both ends of the line, the type of communication is sensor-sharing. The communication of such information is significant not only because the users can feel physically closer to each other, but also because the sensory information obtained at each end of the line (i.e., information which describes not the actor, but the environment of the actor) is shared with the other end (in such cases, the communication is intra-cognitive, despite the fact that the
transferred information describes the environment of the actor, because the environment is treated as a factor which has an implicit, but direct effect on the actor).
is transferred to the other actor using a visual representation consisting of a blinking red dot, or the breath rate of each actor is transferred to the other actor using a visual representation which consists of a discoloration of the screen. The frequency of the discoloration could symbolize the rate of breathing, and the extent of discoloration might symbolize the amount of air inhaled each time. (Similar ideas are investigated in, e.g. [31], [23]). Because the communicating actors are both human, the communication mode is intra-cognitive. Because the sensory modality used to perceive the information (visual system) is different from the modality used to normally perceive the information (it is questionable if such a modality even exists, because people don’t usually feel the pulse or breath rate of other people during normal conversation), we say that the communication is sensor-bridging. The communication of such parameters is significant in that they help further describe the psychological state of the actors. Due to the fact that such parameters are directly imperceptible even in face-to-face communication, the only possibility is to convey them through sensor bridging. In general, the transferred information is considered cognitive because the psychological state of the actors does not depend on this information in a definitive way, but when interpreted by a cognitive system such as a human actor, the information and its context together can help create a deeper understanding of the psychological state of the remote user.
actor, as well as a variety of background scents (using electronic noses and scent generators) to a human actor controlling the robot from a remote teleoperation room. The operating sound of a robot actor can help the teleoperator gain
a good sense of the amount of load the robot is dealing with, how much resistance it is encountering during its operation, etc. Further, the ability to perceive smells from the robot’s surroundings can augment the teleoperator’s perception of possible hazards in the robot’s environment. A further example of inter-cognitive sensor sharing would be the transfer of direct force feedback through e.g. a joystick
. The communication in these examples is inter-cognitive because the robot’s cognitive system is significantly different from the human teleoperator’s cognitive system. Because the transferred information is conveyed directly to the same sensory modality, the communication is also sensor-sharing. Similar to the case of intracognitive sensor-sharing, the transfer of such information is significant because it helps further describe the environment in which the remote cognitive system is operating, which has an implicit effect on the remote cognitive system.
, artificial cognitive systems and the virtual manifestations of these systems (which are gaining wide acceptance in today’s engineering systems, e.g.
as in iSpace [18]) are becoming more and more sophisticated, the operation of these systems and the way in which they organize complex information are, by their nature, essentially
inaccessible in many cases to the human perceptual system and the information representation it uses. For this reason, intercognitive sensor bridging is perhaps the most complex area of CogInfoCom, because it relies on a sophisticated combination of a number of fields from information engineering
and infocommunications to the various cognitive sciences.
A rudimentary example of inter-cognitive sensor-bridging communication that is already in wide use today is the collision-detection system available in many cars which plays a frequency modulated signal, the frequency of which depends on the distance between the car and the (otherwise completely visible) car behind it. In this case, auditory signals are used to convey spatial (visual) information. A further example could be the use of the vibrotactile system to provide force feedback through axial vibrations (this is a commonly adopted approach in various applications, from remote vehicle guidance to telesurgery, e.g. [44], [45], [46], [47], [48]). Force feedback through axial vibration is also very widespread in gaming
, because with practice, the players can easily adapt to the signals and will really interpret them as if they corresponded to a real collision with an object or someone else’s body [49], [50]. It is important to note, however, that the use of vibrations is no longer limited to the transfer of information on collisions or other simple events, but is also used to communicate more complex information, such as warning signals to alert the user’s attention to events whose occurrence is deduced from a combination of events with a more complex structure (e.g., vibrations of smart phones to alert the user of suspicious account activity, etc.). Such event-detection systems can be powerful when combined with iSpace [28].
Finally, more complex examples of sensor bridging in inter-cognitive communication might include the use of electrotactile arrays placed on the tongue to convey visual information received from a camera placed on the forehead (as in [21]), or the transfer of a robot actor’s tactile percepts (as detected by e.g. a laser profilometer) using abstract sounds on the other end of the communication line. In [51], relatively short audio signals (i.e., 2–3 seconds long) are used to convey abstract tactile dimensions such as the softness, roughness, stickiness and temperature of surfaces. The necessity of haptic feedback in virtual environments cannot be underrated [52].
The type of information conveyed through sensor bridging, the extent to which this information is abstract and the sensory modality to which it is conveyed is open to research. As researchers obtain closer estimates to the number of dimensions each sensory modality is sensitive to, and the resolution and speed of each modality’s information processing capabilities, research and development in sensory substitution will surely provide tremendous improvements to today’s engineering systems.
and heart rate
are also examples of sensor-sharing inter-cognitive communication (they are intercognitive, because the transmission of information is effected through the communication links with the robot, and they are sensor-sharing, because they are presented in the same graphical form in which blood pressure and heart rhythm information are normally displayed). At the same time, information from various sensors on the telesurgical robot might be transmitted to a different sensory modality of the teleoperator (e.g., information from moisture sensors using pressure applied to the arm, etc.), which would serve to augment the telesurgeon’s cognitive awareness of the remote environment, and can be considered as sensor-bridging communication resulting in an augmented form of telepresence. Through the intracognitive mode of communication, the two teleoperators can obtain information on each other’s psychological state and environment. Here we can also imagine both distance and sensor-bridging types of communication, all of which can directly or indirectly help raise each telesurgeon’s attention to possible problems or abnormalities the other telesurgeon is experiencing.
Cognitive science
Cognitive science is the interdisciplinary scientific study of mind and its processes. It examines what cognition is, what it does and how it works. It includes research on how information is processed , represented, and transformed in behaviour, nervous system or machine...
s, as well as the various engineering applications which have emerged as the synergic combination of these sciences.
The primary goal of CogInfoCom is to provide a systematic view of how cognitive processes can co-evolve with infocommunications devices so that the capabilities of the human brain
Human brain
The human brain has the same general structure as the brains of other mammals, but is over three times larger than the brain of a typical mammal with an equivalent body size. Estimates for the number of neurons in the human brain range from 80 to 120 billion...
may not only be extended through these devices, irrespective of geographical distance, but may also interact with the capabilities of any artificially cognitive system. This merging and extension of cognitive capabilities is targeted towards engineering applications in which artificial and/or natural cognitive systems are enabled to work together more effectively.
We define two important dimensions of cognitive infocommunications: the mode of communication and the type of communication.
The mode of communication refers to the actors at the two endpoints of communication:
- Intracognitive communication: The mode of communication is intra-cognitive when information transferInformation transferIn telecommunications, information transfer is the process of moving messages containing user information from a source to a sink.Note: The information transfer rate may or may not be equal to the transmission modulation rate.-See also:...
occurs between two cognitive beings with equivalent cognitive capabilities (e.g.: between two humans). - Intercognitive communication: The mode of communication is inter-cognitive when information transfer occurs between two cognitive beings with different cognitive capabilities (e.g.: between a human and an artificially cognitive system).
The type of communication refers to the type of information that is conveyed between the two communicating entities, and the way in which it is carried out:
- Sensor-sharing communication: entities on both ends use the same sensory modality to perceive the communicated information.
- Sensor-bridging communication: sensory information obtained or experienced by each of the entities is not only transmitted, but also transformed to an appropriate and different sensory modality.
- Representation-sharing communication: sensory information obtained or experienced by each of the entities is merely transferred, hence, the same information representation is used on both ends.
- Representation-bridging communication: sensory information transferred to the receiver entity is filtered and/or adapted so that a different information representation is used on the two ends.
Remarks
- Dominant parts of a number of basic ideas behind CogInfoCom are not new in the sense that different aspects of many key points have appeared, and are being actively researched, in several existing areas of IT (e.g. affective computingAffective computingAffective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. It is an interdisciplinary field spanning computer sciences, psychology, and cognitive science...
, human–computer interactionHuman–computer interactionHuman–computer Interaction is the study, planning, and design of the interaction between people and computers. It is often regarded as the intersection of computer science, behavioral sciences, design and several other fields of study...
, human–robot interaction, sensory substitutionSensory substitutionSensory substitution means to transform the characteristics of one sensory modality into stimuli of another sensory modality. It is hoped that sensory substitution systems can help handicapped people by restoring their ability to perceive a certain defective sensory modality by using sensory...
, sensorimotor extension, iSpace research, interactive systems engineeringInteractive Systems EngineeringInteractive systems engineering is considered as an interdisciplinary field with computer engineering, systems engineering, interaction design, software development, aesthetic, ethnography, psychology and usability factors involved...
, ergonomicsErgonomicsErgonomics is the study of designing equipment and devices that fit the human body, its movements, and its cognitive abilities.The International Ergonomics Association defines ergonomics as follows:...
, etc.) - CogInfoCom should not be confused with computational neuroscienceComputational neuroscienceComputational neuroscience is the study of brain function in terms of the information processing properties of the structures that make up the nervous system...
or computational cognitive modeling, which can mainly be considered as a very important set of modeling tools for cognitive sciences (thus indirectly for CogInfoCom), but have no intention of directly serving engineering systems. - A sensor-sharing application of CogInfoCom is novel in the sense that it extends traditional infocommunications by conveying any kind of signal normally perceptible through the actor’s senses to the other end of the communication line. The transferred information may describe not only the actor involved in the communication, but also the environment in which the actor is located. The key determinant of sensor-sharing communication is that the same sensory modality is used to perceive the sensory information on both ends of the infocommunications line.
- Sensor bridging can be taken to mean not only the way in which the information is conveyed (i.e., by changing sensory modality), but also the kind of information that is conveyed. Whenever the transferred information type is imperceptible to the receiving end (e.g., because its cognitive system is incompatible with the information type) the communication of information will necessarily occur through sensor bridging.
- It is clear that a sensor-sharing type of communication can be either representation-sharing or representation-bridging as well, but in the case of sensor-bridging communication, the question may arise whether it is possible for the representation of information to be the same across different modalities. We remark that future research may provide a rationale to equate represenations even across modalities.
The first draft definition of CogInfoCom was given in [1]. The above definition was finalized based on the paper with the joint participation of the Startup Committee and the participants of the 1st International Workshop on Cognitive Infocommunications, held in Tokyo
Tokyo
, ; officially , is one of the 47 prefectures of Japan. Tokyo is the capital of Japan, the center of the Greater Tokyo Area, and the largest metropolitan area of Japan. It is the seat of the Japanese government and the Imperial Palace, and the home of the Japanese Imperial Family...
, Japan
Japan
Japan is an island nation in East Asia. Located in the Pacific Ocean, it lies to the east of the Sea of Japan, China, North Korea, South Korea and Russia, stretching from the Sea of Okhotsk in the north to the East China Sea and Taiwan in the south...
in 2010.
Research Background of Cognitive Infocommunications
The term cognitive infocommunication was used in several papers before the definition was created (e.g., [2], [3], [4]). The definition was created based on the research described in these papers, and was also motivated by work in Intelligent Space [18], [24]. The 1st International Workshop on Cognitive Infocommunications was held in Tokyo, Japan in 2010.The idea that the information systems we use need to be accommodated with artificially cognitive functionalities has culminated in the creation of cognitive informatics [5], [6].
With the strong support of cognitive science, results in cognitive informatics are contributing to the creation of more and more sophisticated artificially cognitive engineering systems. Given this trend, it is rapidly becoming clear that the amount of cognitive content handled by our engineering systems is reaching a point where the communication forms necessary to enable interaction with this content are becoming more and more complex [7], [8], [9], [10], [11].
The inspiration to create engineering systems capable of communicating with users in natural ways is not new. It is one of the primary goals of affective computing to endow information systems with emotion
Emotion
Emotion is a complex psychophysiological experience of an individual's state of mind as interacting with biochemical and environmental influences. In humans, emotion fundamentally involves "physiological arousal, expressive behaviors, and conscious experience." Emotion is associated with mood,...
, and to enable them to communicate these emotions in ways that resonate with the human users [12], [13]. Augmented cognition deals with modeling human attention and using results to create engineering systems which are capable of adapting based on human cognitive skills and momentary attention [14]. In addition, there are a host of research fields which concentrate less on modeling (human) psychological emotion, but aim to allow users to have a more tractable and pleasurable interaction with machines (e.g., human computer interaction, human robot interaction, interactive systems engineering) [15], [16]. Further fields specialize in the communication of hidden parameters to remote environments (e.g., sensory substitution and sensorimotor extension in engineering applications, iSpace research, multimodal interaction) [17], [18], [19], [20], [21], [22], [23].
In recent years, several applications have appeared in the technical literature which combine various aspects of the previously mentioned fields, but also extend them in significant
ways (e.g., [24], [25], [26], [27], [28], [29], [30], [23], [31], [32], [18], [33]). However, in these works, the fact that a new research field is emerging is only implicitly mentioned.
Discussion
The research areas treated within CogInfoCom from three different points of view: the research historical view, the virtual reality point of view and the cognitive informatics view.Historical view
Traditionally, the research fields of informaticsInformatics (academic field)
Informatics is the science of information, the practice of information processing, and the engineering of information systems. Informatics studies the structure, algorithms, behavior, and interactions of natural and artificial systems that store, process, access and communicate information...
, media
Mass media
Mass media refers collectively to all media technologies which are intended to reach a large audience via mass communication. Broadcast media transmit their information electronically and comprise of television, film and radio, movies, CDs, DVDs and some other gadgets like cameras or video consoles...
, and communications were very different areas, treated by researchers from significantly different backgrounds. As a synthesis between pairs of these 3 disciplines, the fields of infocommunications, media informatics and media communications emerged in the latter half of the 20th century. The past evolution of these disciplines points towards their convergence in the near future, given that modern network services aim to provide a more holistic user experience, which presupposes achievements from these different fields [34], [35]. In place of these research areas, with the enormous growth in scope and generality of cognitive sciences in the past few decades, the new fields of cognitive media [36], [37], cognitive informatics and cognitive communications [38],[39] are gradually emerging. In a way analogous to the evolution of infocommunications, media informatics and media communications, we are seeing more and more examples of research achievements which can be categorized as results in cognitive infocommunications, cognitive media informatics and cognitive media communications, even if – as of yet – these fields are not always clearly defined [40], [41], [24],
[25], [29], [30], [42], [18], [43], [36].
The primary goal of CogInfoCom is to use information theoretical methods to synthesize research results in some of these areas, while aiming primarily to make use of these synthesized results in the design of engineering systems. It is novel in the sense that it views both the medium used for communication and the media which is communicated as entities which are interpreted by a cognitive system.
CogInfoCom from a Virtual Reality Perspective
In everyday human-machine interaction, both the human and the machine are located in the same physical space and thus the human can rely onhis / her natural cognitive system to communicate with the machine. In contrast, when 3D virtualization comes into play, the human is no
longer in direct connection with the machine; instead, he / she must communicate with its virtual representation. Thus, the problem of
human-machine interaction is transformed into a problem of human-virtual machine interaction. In such scenarios, the natural communication capabilities of the human become limited due to the restricted interface provided by the virtual representation of the machine (for instance, while the senses of vision and audition still receive considerable amount of information, the tactile and olfactory senses are almost completely restricted in virtual environments, i.e. it is usually not possible to touch or smell the virtual representation of a machine). For this reason, it becomes necessary to develop a virtual cognitive system which can extend the natural one so as to allow the human to effectively communicate with the virtual representation, which essentially becomes an infocommunication system in this extended scenario. One of the primary goals of cognitive infocommunications is to address this problem so that sensory modalities other than the ones normally used in virtual reality research can be employed for the perception of a large variety of information on the virtualized system.
Cognitive Informatics View
Cognitive informatics (CI) is a research field which was created in the early 21st century, and which pioneered the adoption of research results in cognitive sciences within information technologyInformation technology
Information technology is the acquisition, processing, storage and dissemination of vocal, pictorial, textual and numerical information by a microelectronics-based combination of computing and telecommunications...
[5], [6]. The main purpose of CI is to investigate the internal information storing and processing mechanisms in natural intelligent systems such as the human
brain.Much like CogInfoCom, CI also aims primarily to create numerically tractable models which are well grounded from an information theoretical point of view, and are amenable to engineering systems. The key difference between CI and CogInfoCom is that while the results of CI largely converge towards and support the creation of artificial cognitive systems, the goal of CogInfoCom is to enable these systems to communicate with each other and their users efficiently.
Thus, CogInfoCom builds on a large part of results in CI, since it deals with the communication space between the human cognitive system and other natural or artificial cognitive systems.
Examples
Examples are categorized into 4 categories based on all possible combinations of the modes of communication and the two types of communication pertaining to the sensory modalities used during communication.Intra-cognitive sensor-sharing communication
An example of intra-cognitive sensor-sharing communication is when two humans communicate through SkypeSkype
Skype is a software application that allows users to make voice and video calls and chat over the Internet. Calls to other users within the Skype service are free, while calls to both traditional landline telephones and mobile phones can be made for a fee using a debit-based user account system...
or some other telecommunication system, and a large variety of information types (e.g. metalinguistic information and background noises through sound, gesture-based metacommunication through video, etc.) are communicated to both ends of the line. In more futuristic applications, information from other sensory modalities (e.g. smells through the use of electronic nose
Electronic nose
An electronic nose is a device intended to detect odors or flavors.Over the last decade, “electronic sensing” or “e-sensing” technologies have undergone important developments from a technical and commercial point of view...
s and scent generators, tastes using equipment not yet available today) may also be communicated. Because the communicating actors are both human, the communication mode is intra-cognitive, and because the communicated information is shared using the same cognitive subsystems (i.e., the same sensory modalities) on both ends of the line, the type of communication is sensor-sharing. The communication of such information is significant not only because the users can feel physically closer to each other, but also because the sensory information obtained at each end of the line (i.e., information which describes not the actor, but the environment of the actor) is shared with the other end (in such cases, the communication is intra-cognitive, despite the fact that the
transferred information describes the environment of the actor, because the environment is treated as a factor which has an implicit, but direct effect on the actor).
Intra-cognitive sensor-bridging communication
An example of intra-cognitive sensor-bridging communication is when two humans communicate through Skype or some other telecommunication system, and each actor’s pulsePulse
In medicine, one's pulse represents the tactile arterial palpation of the heartbeat by trained fingertips. The pulse may be palpated in any place that allows an artery to be compressed against a bone, such as at the neck , at the wrist , behind the knee , on the inside of the elbow , and near the...
is transferred to the other actor using a visual representation consisting of a blinking red dot, or the breath rate of each actor is transferred to the other actor using a visual representation which consists of a discoloration of the screen. The frequency of the discoloration could symbolize the rate of breathing, and the extent of discoloration might symbolize the amount of air inhaled each time. (Similar ideas are investigated in, e.g. [31], [23]). Because the communicating actors are both human, the communication mode is intra-cognitive. Because the sensory modality used to perceive the information (visual system) is different from the modality used to normally perceive the information (it is questionable if such a modality even exists, because people don’t usually feel the pulse or breath rate of other people during normal conversation), we say that the communication is sensor-bridging. The communication of such parameters is significant in that they help further describe the psychological state of the actors. Due to the fact that such parameters are directly imperceptible even in face-to-face communication, the only possibility is to convey them through sensor bridging. In general, the transferred information is considered cognitive because the psychological state of the actors does not depend on this information in a definitive way, but when interpreted by a cognitive system such as a human actor, the information and its context together can help create a deeper understanding of the psychological state of the remote user.
Inter-cognitive sensor-sharing communication
An example of inter-cognitive sensor-sharing communication might include the transfer of the operating sound of a robotRobot
A robot is a mechanical or virtual intelligent agent that can perform tasks automatically or with guidance, typically by remote control. In practice a robot is usually an electro-mechanical machine that is guided by computer and electronic programming. Robots can be autonomous, semi-autonomous or...
actor, as well as a variety of background scents (using electronic noses and scent generators) to a human actor controlling the robot from a remote teleoperation room. The operating sound of a robot actor can help the teleoperator gain
a good sense of the amount of load the robot is dealing with, how much resistance it is encountering during its operation, etc. Further, the ability to perceive smells from the robot’s surroundings can augment the teleoperator’s perception of possible hazards in the robot’s environment. A further example of inter-cognitive sensor sharing would be the transfer of direct force feedback through e.g. a joystick
Joystick
A joystick is an input device consisting of a stick that pivots on a base and reports its angle or direction to the device it is controlling. Joysticks, also known as 'control columns', are the principal control in the cockpit of many civilian and military aircraft, either as a center stick or...
. The communication in these examples is inter-cognitive because the robot’s cognitive system is significantly different from the human teleoperator’s cognitive system. Because the transferred information is conveyed directly to the same sensory modality, the communication is also sensor-sharing. Similar to the case of intracognitive sensor-sharing, the transfer of such information is significant because it helps further describe the environment in which the remote cognitive system is operating, which has an implicit effect on the remote cognitive system.
Inter-cognitive sensor-bridging communication
As the information systemsInformation systems
Information Systems is an academic/professional discipline bridging the business field and the well-defined computer science field that is evolving toward a new scientific area of study...
, artificial cognitive systems and the virtual manifestations of these systems (which are gaining wide acceptance in today’s engineering systems, e.g.
as in iSpace [18]) are becoming more and more sophisticated, the operation of these systems and the way in which they organize complex information are, by their nature, essentially
inaccessible in many cases to the human perceptual system and the information representation it uses. For this reason, intercognitive sensor bridging is perhaps the most complex area of CogInfoCom, because it relies on a sophisticated combination of a number of fields from information engineering
Information engineering
Information engineering or information engineering methodology in software engineering is an approach to designing and developing information systems.-Overview:...
and infocommunications to the various cognitive sciences.
A rudimentary example of inter-cognitive sensor-bridging communication that is already in wide use today is the collision-detection system available in many cars which plays a frequency modulated signal, the frequency of which depends on the distance between the car and the (otherwise completely visible) car behind it. In this case, auditory signals are used to convey spatial (visual) information. A further example could be the use of the vibrotactile system to provide force feedback through axial vibrations (this is a commonly adopted approach in various applications, from remote vehicle guidance to telesurgery, e.g. [44], [45], [46], [47], [48]). Force feedback through axial vibration is also very widespread in gaming
Gaming
Gaming is the act of playing a game.Gaming may also refer to:* A synonym for Gambling; playing games of chance for money.* Playing a role-playing game, in which players assume fictional roles....
, because with practice, the players can easily adapt to the signals and will really interpret them as if they corresponded to a real collision with an object or someone else’s body [49], [50]. It is important to note, however, that the use of vibrations is no longer limited to the transfer of information on collisions or other simple events, but is also used to communicate more complex information, such as warning signals to alert the user’s attention to events whose occurrence is deduced from a combination of events with a more complex structure (e.g., vibrations of smart phones to alert the user of suspicious account activity, etc.). Such event-detection systems can be powerful when combined with iSpace [28].
Finally, more complex examples of sensor bridging in inter-cognitive communication might include the use of electrotactile arrays placed on the tongue to convey visual information received from a camera placed on the forehead (as in [21]), or the transfer of a robot actor’s tactile percepts (as detected by e.g. a laser profilometer) using abstract sounds on the other end of the communication line. In [51], relatively short audio signals (i.e., 2–3 seconds long) are used to convey abstract tactile dimensions such as the softness, roughness, stickiness and temperature of surfaces. The necessity of haptic feedback in virtual environments cannot be underrated [52].
The type of information conveyed through sensor bridging, the extent to which this information is abstract and the sensory modality to which it is conveyed is open to research. As researchers obtain closer estimates to the number of dimensions each sensory modality is sensitive to, and the resolution and speed of each modality’s information processing capabilities, research and development in sensory substitution will surely provide tremendous improvements to today’s engineering systems.
Complex example
Let us consider a scenario where a telesurgeon in location A is communicating with a telesurgical robot in remote location B, and another telesurgeon in remote location C. At the same time, let us imagine that the other telesurgeon (in location C) is communicating with a different telesurgical robot, also in remote location B (in much the same way as a surgical assistant would perform a different task on the same patient), and the first telesurgeon in location A. In this case, both teleoperators are involved in one channel of inter-cognitive and one channel of intra-cognitive communication. Within these two modes, examples of sensor sharing and sensor bridging might occur at the same time. Each telesurgeon may see a camera view of the robot they are controlling, feel the limits of their motions through direct force feedback, and hear the soft, but whining sound of the operating robot through direct sound transfer. These are all examples of sensor-sharing intercognitive communication. The transmission of the operated patient’s blood pressureBlood pressure
Blood pressure is the pressure exerted by circulating blood upon the walls of blood vessels, and is one of the principal vital signs. When used without further specification, "blood pressure" usually refers to the arterial pressure of the systemic circulation. During each heartbeat, BP varies...
and heart rate
Heart rate
Heart rate is the number of heartbeats per unit of time, typically expressed as beats per minute . Heart rate can vary as the body's need to absorb oxygen and excrete carbon dioxide changes, such as during exercise or sleep....
are also examples of sensor-sharing inter-cognitive communication (they are intercognitive, because the transmission of information is effected through the communication links with the robot, and they are sensor-sharing, because they are presented in the same graphical form in which blood pressure and heart rhythm information are normally displayed). At the same time, information from various sensors on the telesurgical robot might be transmitted to a different sensory modality of the teleoperator (e.g., information from moisture sensors using pressure applied to the arm, etc.), which would serve to augment the telesurgeon’s cognitive awareness of the remote environment, and can be considered as sensor-bridging communication resulting in an augmented form of telepresence. Through the intracognitive mode of communication, the two teleoperators can obtain information on each other’s psychological state and environment. Here we can also imagine both distance and sensor-bridging types of communication, all of which can directly or indirectly help raise each telesurgeon’s attention to possible problems or abnormalities the other telesurgeon is experiencing.