Digital compositing
Encyclopedia
Digital compositing is the process of digitally assembling multiple images to make a final image, typically for print, motion pictures
or screen display. It is the evolution into the digital realm of optical film compositing
.
values that end up a single output pixel.
Consider three pixels;
and
Then, considering all three colour channels, and assuming that the colour channels are expressed in a γ=1 colour space (that is to say, the measured values are proportional to light intensity), we have:
Note that if the operations are performed in a colour space where γ is not equal to 1 then the operation will lead to non-linear effects which can potentially be seen as aliasing
artifacts (or 'jaggies
') along sharp edges in the matte. More generally, nonlinear compositing can have effects such as "halos" around composited objects, because the influence of the alpha channel is non-linear. It is possible for a compositing artist to compensate for the effects of compositing in non-linear space.
Performing alpha blending is an expensive operation if performed on an entire image or 3D scene. If this operation has to be done in real time video games there is an easy trick to boost performance.
By simply rewriting the mathematical expression one can save 50% of the multiplications required.
Consider the case when we have four layers to blend to produce the final image: F=A*(B*(C*D)) where A, B, C, D are partially transparent image layers and "*" denotes a compositing operator (with the left layer on top of the right layer). If only layer C changes, we should find a way to avoid re-blending all of the layers when computing F. Without any special considerations, four full-image blends would need to occur. For compositing operators that are commutative, such as additive blending, it is safe to re-order the blending operations. In this case, we might compute T=A*(B*D) only once and simply blend T*C to produce F, a single operation. Unfortunately, most operators are not commutative. However, many are associative, suggesting it is safe to re-group operations without changing their order. In this case we may compute S=A*B once and save this result. To form F with an associative operator, we need only do two additional compositing operations to integrate the new layer C: F=S*(C*D). Note that this expression indicates compositing C with all of the layers below it in one step and then blending all of the layers on top of it with the previous result to produce the final image in the second step.
If all layers of an image change regularly but a large number of layer still need to be composited (such as in distributed rendering), the commutativity of a compositing operator can still be exploited to speed up computation through parallelism even when there is no gain from pre-computation. Again, consider the image F=A*(B*(C*D)). Each compositing operation in this expression depends on the next, leading to serial computation. However, commutativity can allow us to rewrite F=(A*B)*(C*D) where there are clearly two operations that do not depend on each other that may be executed in parallel. In general, we can build a tree of pair-wise compositing operations with a height that is logarithmic in the number of layers.
, which operated in a logarithmic color space, which more closely mimics the natural light response of film emulsions (the Cineon system, made by Kodak, is no longer in production). Due to the limitations of processing speed and memory, compositing artists did not usually have the luxury of having the system make intermediate conversions to linear space for the compositing steps. Over time, the limitations have become much less significant, and now most compositing is done in a linear color space, even in cases where the source imagery is in a logarithmic color space.
Compositing often also includes scaling, retouching and colour correction of images.
Node-based compositing represents an entire composite as a tree graph, linking media objects and effects in a procedural map, intuitively laying out the progression from source input to final output, and is in fact the way all compositing applications internally handle composites. This type of compositing interface allows great flexibility, including the ability to modify the parameters of an earlier image processing step "in context" (while viewing the final composite
). Node-based compositing packages often handle keyframing and time effects poorly, as their workflow does not stem directly from a timeline, as do layer-based compositing packages. Software which incorporates a node based interface include Apple Shake, Blender
, eyeon Fusion
, and Foundry Nuke
.
Layer-based compositing represents each media object in a composite as a separate layer within a timeline, each with its own time bounds, effects, and keyframes. All the layers are stacked, one above the next, in any desired order; and the bottom layer is usually rendered
as a base in the resultant image, with each higher layer being progressively rendered on top of the previously composited of layers, moving upward until all layers have been rendered into the final composite. Layer-based compositing is very well suited for rapid 2D
and limited 3D
effects such as in motion graphics, but becomes awkward for more complex composites entailing a large number of layers. A partial solution to this is some programs' ability to view the composite-order of elements (such as images, effects, or other attributes) with a visual diagram called a flowchart to nest compositions
, or "comps," directly into other compositions, thereby adding complexity to the render-order by first compositing layers in the beginning composition, then combining that resultant image with the layered images from the proceeding composition, and so on. An example of this exists in the Adobe program After Effects.
Film
A film, also called a movie or motion picture, is a series of still or moving images. It is produced by recording photographic images with cameras, or by creating images using animation techniques or visual effects...
or screen display. It is the evolution into the digital realm of optical film compositing
Compositing
Compositing is the combining of visual elements from separate sources into single images, often to create the illusion that all those elements are parts of the same scene. Live-action shooting for compositing is variously called "chroma key", "blue screen", "green screen" and other names. Today,...
.
Mathematics
The basic operation used is known as 'alpha blending', where an opacity value, 'α' is used to control the proportions of two input pixelPixel
In digital imaging, a pixel, or pel, is a single point in a raster image, or the smallest addressable screen element in a display device; it is the smallest unit of picture that can be represented or controlled....
values that end up a single output pixel.
Consider three pixels;
- a foreground pixel, f
- a background pixel, b
- a composited pixel, c
and
- α, the opacity value of the foreground pixel. (α=1 for opaque foreground, α=0 for a completely transparent foreground). A monochrome raster image where the pixel values are to be interpreted as alpha values is known as a matteMatte (filmmaking)Mattes are used in photography and special effects filmmaking to combine two or more image elements into a single, final image. Usually, mattes are used to combine a foreground image with a background image . In this case, the matte is the background painting...
.
Then, considering all three colour channels, and assuming that the colour channels are expressed in a γ=1 colour space (that is to say, the measured values are proportional to light intensity), we have:
- cr = α fr + (1 − α) br
- cg = α fg + (1 − α) bg
- cb = α fb + (1 − α) bb
Note that if the operations are performed in a colour space where γ is not equal to 1 then the operation will lead to non-linear effects which can potentially be seen as aliasing
Aliasing
In signal processing and related disciplines, aliasing refers to an effect that causes different signals to become indistinguishable when sampled...
artifacts (or 'jaggies
Jaggies
"Jaggies" is the informal name for artifacts in raster images, most frequently from aliasing, which in turn is often caused by non-linear mixing effects producing high-frequency components and/or missing or poor anti-aliasing filtering prior to sampling....
') along sharp edges in the matte. More generally, nonlinear compositing can have effects such as "halos" around composited objects, because the influence of the alpha channel is non-linear. It is possible for a compositing artist to compensate for the effects of compositing in non-linear space.
Performing alpha blending is an expensive operation if performed on an entire image or 3D scene. If this operation has to be done in real time video games there is an easy trick to boost performance.
- cout = α fin + (1 − α) bin
- cout = α fin + bin − α bin
- cout = bin + α (fin − bin)
By simply rewriting the mathematical expression one can save 50% of the multiplications required.
Algebraic properties
When many partially transparent layers need to be composited together, it is worthwhile to consider the algebraic properties of compositing operators used. Specifically, the associativity and commutativity determine when repeated calculation can or cannot be avoided.Consider the case when we have four layers to blend to produce the final image: F=A*(B*(C*D)) where A, B, C, D are partially transparent image layers and "*" denotes a compositing operator (with the left layer on top of the right layer). If only layer C changes, we should find a way to avoid re-blending all of the layers when computing F. Without any special considerations, four full-image blends would need to occur. For compositing operators that are commutative, such as additive blending, it is safe to re-order the blending operations. In this case, we might compute T=A*(B*D) only once and simply blend T*C to produce F, a single operation. Unfortunately, most operators are not commutative. However, many are associative, suggesting it is safe to re-group operations without changing their order. In this case we may compute S=A*B once and save this result. To form F with an associative operator, we need only do two additional compositing operations to integrate the new layer C: F=S*(C*D). Note that this expression indicates compositing C with all of the layers below it in one step and then blending all of the layers on top of it with the previous result to produce the final image in the second step.
If all layers of an image change regularly but a large number of layer still need to be composited (such as in distributed rendering), the commutativity of a compositing operator can still be exploited to speed up computation through parallelism even when there is no gain from pre-computation. Again, consider the image F=A*(B*(C*D)). Each compositing operation in this expression depends on the next, leading to serial computation. However, commutativity can allow us to rewrite F=(A*B)*(C*D) where there are clearly two operations that do not depend on each other that may be executed in parallel. In general, we can build a tree of pair-wise compositing operations with a height that is logarithmic in the number of layers.
Software
The most historically significant nonlinear compositing system was the CineonCineon
Cineon System was a computer based system, integrating a film scanner and recorder hardware, digital software for compositing and effects, image restoration, color management and the development of a proprietary file format designed by Kodak. These components were created to support the work flow...
, which operated in a logarithmic color space, which more closely mimics the natural light response of film emulsions (the Cineon system, made by Kodak, is no longer in production). Due to the limitations of processing speed and memory, compositing artists did not usually have the luxury of having the system make intermediate conversions to linear space for the compositing steps. Over time, the limitations have become much less significant, and now most compositing is done in a linear color space, even in cases where the source imagery is in a logarithmic color space.
Compositing often also includes scaling, retouching and colour correction of images.
Node-based and layer-based compositing
There are two radically different digital compositing workflows: node-based compositing and layer-based compositing.Node-based compositing represents an entire composite as a tree graph, linking media objects and effects in a procedural map, intuitively laying out the progression from source input to final output, and is in fact the way all compositing applications internally handle composites. This type of compositing interface allows great flexibility, including the ability to modify the parameters of an earlier image processing step "in context" (while viewing the final composite
Compositing
Compositing is the combining of visual elements from separate sources into single images, often to create the illusion that all those elements are parts of the same scene. Live-action shooting for compositing is variously called "chroma key", "blue screen", "green screen" and other names. Today,...
). Node-based compositing packages often handle keyframing and time effects poorly, as their workflow does not stem directly from a timeline, as do layer-based compositing packages. Software which incorporates a node based interface include Apple Shake, Blender
Blender (software)
Blender is a free and open-source 3D computer graphics software product used for creating animated films, visual effects, interactive 3D applications or video games. The current release version is 2.60, and was released on October 19, 2011...
, eyeon Fusion
Eyeon Fusion
Fusion is an image compositing software program created by eyeon Software Inc. It is typically used to create visual effects and digital compositing for film, HD and commercials...
, and Foundry Nuke
Nuke (software)
Nuke is a node-based compositor produced by The Foundry, and used for film and television post-production. Nuke is cross-platform, and is available for Microsoft Windows, Mac OS X, and Linux...
.
Layer-based compositing represents each media object in a composite as a separate layer within a timeline, each with its own time bounds, effects, and keyframes. All the layers are stacked, one above the next, in any desired order; and the bottom layer is usually rendered
Render
Render or rendering may refer to:In Computing:* Artistic rendering, creating, shading and texturing of an image* Architectural rendering, creating two-dimensional images or animations showing the attributes of a proposed architectural design...
as a base in the resultant image, with each higher layer being progressively rendered on top of the previously composited of layers, moving upward until all layers have been rendered into the final composite. Layer-based compositing is very well suited for rapid 2D
2D computer graphics
2D computer graphics is the computer-based generation of digital images—mostly from two-dimensional models and by techniques specific to them...
and limited 3D
3D computer graphics
3D computer graphics are graphics that use a three-dimensional representation of geometric data that is stored in the computer for the purposes of performing calculations and rendering 2D images...
effects such as in motion graphics, but becomes awkward for more complex composites entailing a large number of layers. A partial solution to this is some programs' ability to view the composite-order of elements (such as images, effects, or other attributes) with a visual diagram called a flowchart to nest compositions
Comp
Comp may refer to:In music:*Accompaniment, especially in jazz *Compilation album*Comp In business and finance:*Comparable company analysis*Comparables, in real estate*Same Store Sales...
, or "comps," directly into other compositions, thereby adding complexity to the render-order by first compositing layers in the beginning composition, then combining that resultant image with the layered images from the proceeding composition, and so on. An example of this exists in the Adobe program After Effects.
- Adobe After EffectsAdobe After EffectsAdobe After Effects is a digital motion graphics and compositing software published by Adobe Systems, used in the post-production process of filmmaking and television production. Its main uses are the origination of 2D and 2.5D animation, visual effects compositing and finishing...
- Apple MotionApple MotionMotion is a software application produced by Apple Inc. for their Mac OS X operating system. It is used to create and edit motion graphics, titling for video production and film production, and 2D and 3D compositing for visual effects.-History:...
- Apple Shake(Discontinued)
- Autodesk Combustion
- Autodesk Flint, Flame & Inferno
- Autodesk Smoke
- Autodesk Softimage Previously Softimage|XSI
- Autodesk ToxikAutodesk ToxikAutodesk Toxik is an interactive node based, film composing solution developed by Autodesk Media and Entertainment, a subsidiary of Autodesk, Inc. - Features :...
- Aviary PeacockAviary (application suite)Aviary is a proprietary web-based multi-media application suite that is free to use and currently under development by Worth1000. They are using the Adobe Flex platform for most of their tools...
- Avid DS
- BlenderBlender (software)Blender is a free and open-source 3D computer graphics software product used for creating animated films, visual effects, interactive 3D applications or video games. The current release version is 2.60, and was released on October 19, 2011...
- Kodak CineonCineonCineon System was a computer based system, integrating a film scanner and recorder hardware, digital software for compositing and effects, image restoration, color management and the development of a proprietary file format designed by Kodak. These components were created to support the work flow...
- CompTime Industrial Light & Magic
- eyeon FusionEyeon FusionFusion is an image compositing software program created by eyeon Software Inc. It is typically used to create visual effects and digital compositing for film, HD and commercials...
- FXHome CompositeLab Pro
- Industrial Light & Magic's proprietary SaberIndustrial Light & Magic's proprietary SaberIndustrial Light and Magic's proprietary Sabre compositing technology is actually based on Discreet Logic's Inferno and Flame. Several key innovations separate it from the 'off the shelf' versions. Most important is the tools that accurately deal with logarithmic color conversions into and out of...
- JahshakaJahshakaCinefx aims to become a cross-platform, open source, free, video editing software, effects, and compositing suite. It is currently in alpha stage, supporting realtime effects rendering, but lacking useful implementations of many features such as the non-linear editing system...
- KdenliveKdenliveKdenlive is an open source video editing software package based on the MLT Framework that focuses on flexibility and ease of use...
- Foundry NukeNuke (software)Nuke is a node-based compositor produced by The Foundry, and used for film and television post-production. Nuke is cross-platform, and is available for Microsoft Windows, Mac OS X, and Linux...
- OpenShot Video EditorOpenShot Video EditorOpenShot Video Editor is open-source, video editing software package for Linux, built with Python, GTK, and the MLT Framework. The project was started in August 2008 by Jonathan Thomas, with the objective to provide a stable, free, and friendly to use video editor.-OpenShot features:* Support for...
- Pinnacle CommotionPinnacle CommotionCommotion is a visual effects application, originally released by Puffin Designs. Puffin Designs was founded by Scott Squires and Forest Key to market Commotion....
- SideFX Houdini Halo (See Houdini (software)Houdini (software)Houdini is a high-end 3D animation package developed by Side Effects Software which is headquartered in Toronto, Canada. It is a rewrite of the PRISMS ecosystem of standalone tools. Its chief distinction from other packages is that it has been designed as a purely procedural environment...
) - Silicon Grail Rayz and Chalice
See also
- Broadcast designerBroadcast designerA broadcast designer is a person involved with creating graphic designs and electronic media incorporated in television productions that are used by character generator operators...
- Chroma keyChroma keyChroma key compositing is a technique for compositing two images together. A color range in the top layer is made transparent, revealing another image behind. The chroma keying technique is commonly used in video production and post-production...
- Digital assetDigital assetA digital asset is any item of text or media that has been formatted into a binary source that includes the right to use it. A digital file without the right to use it is not an asset. Digital assets are categorised in three major groups which may be defined as textual content , images and...
- Digital cinemaDigital cinemaDigital cinema refers to the use of digital technology to distribute and project motion pictures. A movie can be distributed via hard drives, optical disks or satellite and projected using a digital projector instead of a conventional film projector...
- Digital on-screen graphicDigital on-screen graphicA digital on-screen graphic is a watermark-like station logo that many television broadcasters overlay over a portion of the screen-area of their programs to identify the channel...
(BUG) - Gamma correctionGamma correctionGamma correction, gamma nonlinearity, gamma encoding, or often simply gamma, is the name of a nonlinear operation used to code and decode luminance or tristimulus values in video or still image systems...
- Graphics coordinatorGraphics coordinatorA graphics coordinator, GC, or font assist is an individual who works, usually on a television show, as a producer of on-air still and motion graphics. The graphics coordinator decides what content should be displayed on-air - such as on a fullpage or a lower third...
- Motion graphic
- Motion graphic design
Further reading
- T. Porter and T. Duff, "Compositing Digital Images", Proceedings of SIGGRAPH '84, 18 (1984).
- The Art and Science of Digital Compositing (ISBN 0-12-133960-2)