Sonic interaction design
Encyclopedia
Sonic interaction design is the study and exploitation of sound
as one of the principal channels conveying information
, meaning, and aesthetic/emotional qualities in interactive contexts. Sonic interaction design is at the intersection of interaction design
and sound and music computing
. If interaction design
is about designing objects people interact with, and such interactions are facilitated by computational means, in Sonic Interaction Design sound is mediating interaction either as a display of processes or as an input medium.
in this area focuses on experimental scientific findings about human sound reception in interactive
contexts.
During closed-loop interactions, the users manipulate an interface that produces sound, and the sonic feedback affects in turn the users’ manipulation. In other words, there is a tight coupling between auditory perception
and action. Listening to sounds might not only activate a representation of how the sound was made: it might also prepare the listener to react to the sound . Cognitive representations of sounds might be associated with action-planning schemas, and sounds can also unconsciously cue a further reaction on the part of the listener.
Sonic interactions have the potential to influence the users’ emotions
: the quality of the sounds affects the pleasantness of the interaction, and the difficulty of the manipulation influences whether the user feels in control or not.
. Research and development in this area relies on studies from other disciplines, such as:
In design research
for sonic products a set of practices have been inherited from a variety of fields. Such practices have been tested in contexts where research and pedagogy naturally intermix. Among these practices it suffices to mention:
The experience is in the focus, addressing how humans are affected by the sound, and vice versa. Interactive art and music allows us to question existing paradigms and models of how we interact with technology and sound, going beyond paradigms of control (human controlling a machine). Users are part of a loop which includes action and perception.
Interactive art and music projects invite explorative actions and playful engagement. There is also a multi-sensory aspect, especially haptic-audio and audio-visual projects are popular. Amongst many other influences, this field is informed by the development of the roles of instrument-maker, composer and performer merging .
Artistic research in Sonic Interaction Design is about productions in the interactive art
s and performing arts
, exploiting the role of enactive engagement with sound–augmented interactive objects.
is the data-dependent generation of sound, if the transformation is systematic, objective and reproducible, so that it can be used as scientific method .
For Sonic Interaction Design, sonification provides a set of methods to create interaction sounds that encode relevant data, so that the user can perceive or interpret the conveyed information. Sonification does not necessarily need to represent huge amounts of data in sound, but may only convey one or few data values in a sound. To give an example, imagine a light switch that, on activation would create a short sound that depends on the electric power consumed through the cable: more energy-wasting lamps would perhaps systematically result in more annoying switch sounds. This example shows that sonification aims to provide some information by using its systematic transformation into sound.
The integration of data-driven elements in interaction sound may serve different purposes:
Within the field of Sonification
, Sonic Interaction Design acknowledges the importance of human interaction
for understanding and using auditory feedback
. Within Sonic Interaction Design, sonification can help and offer solutions, methods, and techniques to inspire and guide the design of products or interactive systems.
Sound
Sound is a mechanical wave that is an oscillation of pressure transmitted through a solid, liquid, or gas, composed of frequencies within the range of hearing and of a level sufficiently strong to be heard, or the sensation stimulated in organs of hearing by such vibrations.-Propagation of...
as one of the principal channels conveying information
Information
Information in its most restricted technical sense is a message or collection of messages that consists of an ordered sequence of symbols, or it is the meaning that can be interpreted from such a message or collection of messages. Information can be recorded or transmitted. It can be recorded as...
, meaning, and aesthetic/emotional qualities in interactive contexts. Sonic interaction design is at the intersection of interaction design
Interaction design
In design, human–computer interaction, and software development, interaction design, often abbreviated IxD, is "the practice of designing interactive digital products, environments, systems, and services." Like many other design fields interaction design also has an interest in form but its main...
and sound and music computing
Sound and music computing
Sound and Music Computing is a research field that studies the whole sound and music communication chain from a multidisciplinary point of view...
. If interaction design
Interaction design
In design, human–computer interaction, and software development, interaction design, often abbreviated IxD, is "the practice of designing interactive digital products, environments, systems, and services." Like many other design fields interaction design also has an interest in form but its main...
is about designing objects people interact with, and such interactions are facilitated by computational means, in Sonic Interaction Design sound is mediating interaction either as a display of processes or as an input medium.
Perceptual, cognitive, and emotional study of sonic interactions
ResearchResearch
Research can be defined as the scientific search for knowledge, or as any systematic investigation, to establish novel facts, solve new or existing problems, prove new ideas, or develop new theories, usually using a scientific method...
in this area focuses on experimental scientific findings about human sound reception in interactive
Interactivity
In the fields of information science, communication, and industrial design, there is debate over the meaning of interactivity. In the "contingency view" of interactivity, there are three levels:...
contexts.
During closed-loop interactions, the users manipulate an interface that produces sound, and the sonic feedback affects in turn the users’ manipulation. In other words, there is a tight coupling between auditory perception
Perception
Perception is the process of attaining awareness or understanding of the environment by organizing and interpreting sensory information. All perception involves signals in the nervous system, which in turn result from physical stimulation of the sense organs...
and action. Listening to sounds might not only activate a representation of how the sound was made: it might also prepare the listener to react to the sound . Cognitive representations of sounds might be associated with action-planning schemas, and sounds can also unconsciously cue a further reaction on the part of the listener.
Sonic interactions have the potential to influence the users’ emotions
Emotion
Emotion is a complex psychophysiological experience of an individual's state of mind as interacting with biochemical and environmental influences. In humans, emotion fundamentally involves "physiological arousal, expressive behaviors, and conscious experience." Emotion is associated with mood,...
: the quality of the sounds affects the pleasantness of the interaction, and the difficulty of the manipulation influences whether the user feels in control or not.
Product sound design
Product design in the context of Sonic Interaction Design is dealing with methods and experiences for designing interactive products having a salient sonic behaviour. Products, in this context, are either tangible and functional objects that are designed to be manipulated, or usable simulations of such objects as in virtual prototypingVirtual prototyping
Virtual prototyping is a technique in the process of product development. It involves using computer-aided design and computer-aided engineering software to validate a design before committing to making a physical prototype...
. Research and development in this area relies on studies from other disciplines, such as:
- product sound qualitySound qualitySound quality is the quality of the audio output from various electronic devices. Sound quality can be defined as the degree of accuracy with which a device records or emits the original sound waves...
; - acoustic ecologyAcoustic ecologyAcoustic ecology, sometimes called ecoacoustics or soundscape studies, is the relationship, mediated through sound, between living beings and their environment. Acoustic ecology studies started in the late 1960s with R. Murray Schafer and his team at Simon Fraser University as part of the World...
, i.e. the relationship, mediated through sound, between living beings and their environmentEcosystemAn ecosystem is a biological environment consisting of all the organisms living in a particular area, as well as all the nonliving , physical components of the environment with which the organisms interact, such as air, soil, water and sunlight....
; - filmFilmA film, also called a movie or motion picture, is a series of still or moving images. It is produced by recording photographic images with cameras, or by creating images using animation techniques or visual effects...
sound; - Computer and video game sound;
- sound cultureSound cultureSoundCulture is also an international organisation of artists, theorists and performers with a common interest to promote and develop the sonic arts....
, i.e. the study of how the production and consumption of sound have changed throughout history and within different societies.
In design research
Design research
Design research investigates the process of designing in all its many fields. It is thus related to Design methods in general or for particular disciplines. A primary interpretation of design research is that it is concerned with undertaking research into the design process. Secondary...
for sonic products a set of practices have been inherited from a variety of fields. Such practices have been tested in contexts where research and pedagogy naturally intermix. Among these practices it suffices to mention:
- bodystormingBodystormingBodystorming is a technique sometimes used in interaction design or as a creativity technique.The idea is to imagine what it would be like if the product existed, and act as though it exists, ideally in the place it would be used...
, especially when combined with vocal sketching , where participants produce vocal imitations to mimic the sonic behavior of objects while they are being interacted with; - theatrical practices, such as theatrical metaphors and dramatic performance;
- basic design , based on demonstrations and intersubjectivityIntersubjectivityIntersubjectivity is a term used in philosophy, psychology, sociology and anthropology to describe a condition somewhere between subjectivity and objectivity, one in which a phenomenon is personally experienced but by more than one subject....
; - video prototyping with sonic overlays;
- Foley artistFoley artistFoley is the reproduction of everyday sounds for use in filmmaking. These reproduced sounds can be anything from the swishing of clothing and footsteps to squeaky doors and breaking glass. The best foley art is so well integrated into a film that it goes unnoticed by the audience. It helps to...
ry in filmmakingFilmmakingFilmmaking is the process of making a film, from an initial story, idea, or commission, through scriptwriting, casting, shooting, directing, editing, and screening the finished product before an audience that may result in a theatrical release or television program...
; - actingActingActing is the work of an actor or actress, which is a person in theatre, television, film, or any other storytelling medium who tells the story by portraying a character and, usually, speaking or singing the written text or play....
out sound dramas .
Interactive art and music
In the context of Sonic Interaction Design, interactive art and music projects are designing and researching aesthetic experiences where sonic interaction is in the focus. The creative and expressive aspects – the aesthetics – are more important than conveying information through sound. Practices include installations, performances, public art and interactions between humans through digitally-augmented objects/environments. These often integrate elements such as embedded technology, gesture-sensitive devices, speakers or context-aware systems.The experience is in the focus, addressing how humans are affected by the sound, and vice versa. Interactive art and music allows us to question existing paradigms and models of how we interact with technology and sound, going beyond paradigms of control (human controlling a machine). Users are part of a loop which includes action and perception.
Interactive art and music projects invite explorative actions and playful engagement. There is also a multi-sensory aspect, especially haptic-audio and audio-visual projects are popular. Amongst many other influences, this field is informed by the development of the roles of instrument-maker, composer and performer merging .
Artistic research in Sonic Interaction Design is about productions in the interactive art
Interactive art
Interactive art is a form of installation-based art that involves the spectator in a way that allows the art to achieve its purpose. Some installations achieve this by letting the observer or visitor "walk" in, on, and around them; Some others ask the artist to become part of the artwork.Works of...
s and performing arts
Performing arts
The performing arts are those forms art which differ from the plastic arts insofar as the former uses the artist's own body, face, and presence as a medium, and the latter uses materials such as clay, metal or paint which can be molded or transformed to create some physical art object...
, exploiting the role of enactive engagement with sound–augmented interactive objects.
Sonification
SonificationSonification
Sonification, a form of auditory display, is the use of non-speech audio to convey information or perceptualize data. Auditory perception has high temporal and pressure resolution, which opens up possibilities for it as an alternative or complement to visualization techniques.For example, the...
is the data-dependent generation of sound, if the transformation is systematic, objective and reproducible, so that it can be used as scientific method .
For Sonic Interaction Design, sonification provides a set of methods to create interaction sounds that encode relevant data, so that the user can perceive or interpret the conveyed information. Sonification does not necessarily need to represent huge amounts of data in sound, but may only convey one or few data values in a sound. To give an example, imagine a light switch that, on activation would create a short sound that depends on the electric power consumed through the cable: more energy-wasting lamps would perhaps systematically result in more annoying switch sounds. This example shows that sonification aims to provide some information by using its systematic transformation into sound.
The integration of data-driven elements in interaction sound may serve different purposes:
- to allow the users to refine their actions via auditory feedback. Example: the sonification-enhanced drilling machine which indicates by sound when a wanted orientation to the wall is reached.
- to create a sonic gestaltGestaltDie Gestalt is a German word for form or shape. It is used in English to refer to a concept of 'wholeness'. Gestalt may also refer to:* Gestalt psychology , a theory of mind and brain, describing the Gestalt effect....
for the interaction which allows users to compare the detailed performance on repeated interactions: for instance rowing strokes may be sonified so that the sportsmen can better synchronize their action. - to enable novel functions that would otherwise be not available (e.g. a bottle that displays by sound how much fluid is poured into glasses so that the users can more easily fill the equal amount of liquid in different glasses).
- to enhance the awareness of information that matters, by coupling sonifications to interactions (e.g. auditory augmentation of structure-borne sounds, such as keyboard typing clicks, so that data of interest become peripherally perceivable)
Within the field of Sonification
Sonification
Sonification, a form of auditory display, is the use of non-speech audio to convey information or perceptualize data. Auditory perception has high temporal and pressure resolution, which opens up possibilities for it as an alternative or complement to visualization techniques.For example, the...
, Sonic Interaction Design acknowledges the importance of human interaction
Interaction
Interaction is a kind of action that occurs as two or more objects have an effect upon one another. The idea of a two-way effect is essential in the concept of interaction, as opposed to a one-way causal effect...
for understanding and using auditory feedback
Feedback
Feedback describes the situation when output from an event or phenomenon in the past will influence an occurrence or occurrences of the same Feedback describes the situation when output from (or information about the result of) an event or phenomenon in the past will influence an occurrence or...
. Within Sonic Interaction Design, sonification can help and offer solutions, methods, and techniques to inspire and guide the design of products or interactive systems.
See also
|
Interactive Systems Engineering Interactive systems engineering is considered as an interdisciplinary field with computer engineering, systems engineering, interaction design, software development, aesthetic, ethnography, psychology and usability factors involved... Interaction design pattern Interaction design patterns are a way to describe solutions to common usability or accessibility problems in a specific context. They document interaction models that make it easier for users to understand an interface and accomplish their tasks.-History:... 3D Interaction In computing, 3D interaction is a form of human-machine interaction where users are able to move and perform interaction in 3D space. Both human and machine process information where the physical position of elements in the 3D space is relevant.... Human–computer interaction Human–computer Interaction is the study, planning, and design of the interaction between people and computers. It is often regarded as the intersection of computer science, behavioral sciences, design and several other fields of study... Sound and music computing Sound and Music Computing is a research field that studies the whole sound and music communication chain from a multidisciplinary point of view... |
User interface The user interface, in the industrial design field of human–machine interaction, is the space where interaction between humans and machines occurs. The goal of interaction between a human and a machine at the user interface is effective operation and control of the machine, and feedback from the... User interface design User interface design or user interface engineering is the design of computers, appliances, machines, mobile communication devices, software applications, and websites with the focus on the user's experience and interaction... User-centered design In broad terms, user-centered design or pervasive usability is a design philosophy and a process in which the needs, wants, and limitations of end users of a product are given extensive attention at each stage of the design process... Usage-centered design Usage-centered design is an approach to user interface design based on a focus on user intentions and usage patterns. It analyzes users in terms of the roles they play in relation to systems and employs abstract use cases for task analysis... Virtual reality Virtual reality , also known as virtuality, is a term that applies to computer-simulated environments that can simulate physical presence in places in the real world, as well as in imaginary worlds... |
Further reading
- Stefano Delle Monache, Pietro Polotti, Davide Rocchesso (2010). A Toolkit for Explorations in Sonic Interaction Design. In: Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound, 2010, New York (AM '10), ISBN: 978-1-4503-0046-9, doi 10.1145/1859799.1859800, citation
- Eoin Brazil and Mikael Fernström, (2009). Empirically Based Auditory Display Design. In: Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23–25 July 2009, Porto, Portugal. Available: online.
- Karmen Franinović, Yon Visell, Daniel Hug, (2007). Sound Embodied: A Report on Sonic Interaction Design in Everyday Artifacts. In: Proceedings of the 13th International Conference on Auditory Display, Montréal, Canada, June 26–29, 2007. Available: online.
- Ernest A. Edmonds, Alastair Weakley, Linda Candy, Mark Fell, Roger Knott, and Sandra Pauletto, (2005). "The Studio as Laboratory: Combining Creative Practice and Digital Technology Research". International Journal of Man-Machine Studies 63(4–5): 452–481. Available: online.
- Ernest Edmonds, Andrew Martin, and Sandra Pauletto, (2004). Audio-visual Interfaces in Digital Art. In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology (ACE '04), Singapore, June 3–5, 2004, pp. 331–336, . Available: online.
- Thomas Hermann and Andy Hunt, (2004). The Importance of Interaction in Sonification. In: Proceedings of ICAD Tenth Meeting of the International Conference on Auditory Display, Sydney, Australia, July 6–9, 2004. Available: online.
- Niklas Röber and Maic Masuch, (2004). Interacting With Sound: An Interaction Paradigm for Virtual Auditory Worlds. In: Proceedings of ICAD Tenth Meeting of the International Conference on Auditory Display, Sydney, Australia, July 6–9, 2004. Available: online.
External links
|
|