Image stitching
Encyclopedia
Image stitching or photo stitching is the process of combining multiple photograph
ic image
s with overlapping fields of view to produce a segmented panorama
or high-resolution image. Commonly performed through the use of computer software
, most approaches to image stitching require nearly exact overlaps between images and identical exposures to produce seamless results. Some digital camera
s can stitch their photos internally.
The image stitching process can be divided into three main components - image registration, calibration and blending.
Image registration involves matching features in a set of images or using direct alignment methods to search for image alignments that minimize the sum of absolute differences
between overlapping pixels. When using direct alignment methods one might first calibrate one's images to get better results. Additionally, users may input a rough model of the panorama to help the feature matching stage, so that - for example - only neighboring images are searched for matching features. Since there are smaller group of features for matching, the result of the search is more accurate and execution of the comparison is faster.
Image calibration aims to minimize differences between an ideal lens models and the camera-lens combination that was used, optical defects such as distortions, exposure
differences between images, vignetting
, camera response and chromatic aberrations
. If feature detection methods were used to register images and absolute positions of the features were recorded and saved, stitching software may use the data for geometric optimization of the images in addition to placing the images on the panosphere. Panotools and its various derivative programs use this method.
Image blending involves executing the adjustments figured out in the calibration stage, combined with remapping of the images to an output projection. Colors are adjusted
between images to compensate for exposure differences. If applicable, high dynamic range
merging is done along with motion compensation
and deghosting. Images are blended together and seam line adjustment is done to minimize the visibility of seams between images.
s.
Rectilinear projection, where the stitched image is viewed on a two-dimensional plane intersecting the panosphere in a single point. Lines that are straight in reality are shown as straight regardless of their directions on the image. Wide views - around 120° or so - start to exhibit severe distortion near the image borders. One case of rectilinear projection is the use of cube faces with cubic mapping
for panorama viewing. Panorama is mapped to six squares, each cube face showing 90 by 90 degree area of the panorama.
Cylindrical projection, where the stitched image shows a 360° horizontal field of view and a limited vertical field of view. Panoramas in this projection are meant to be viewed as though the image is wrapped into a cylinder and viewed from within. When viewed on a 2D plane, horizontal lines appear curved while vertical lines remain straight. Vertical distortion increases rapidly when nearing the top of the panosphere. There are various other cylindrical formats, such as Mercator
and Miller cylindrical which have less distortion near the poles of the panosphere.
Spherical projection or equirectangular projection
— which is strictly speaking another cylindrical projection — where the stitched image shows a 360° horizontal by 180° vertical field of view i.e the whole sphere. Panoramas in this projection are meant to be viewed as though the image is wrapped into a sphere and viewed from within. When viewed on a 2D plane, horizontal lines appear curved as in a cylindrical projection, while vertical lines remain vertical.
Since a panorama is basically map of a sphere, various other mapping projections from cartographers
can also be used if so desired. Additionally there are specialized projections which may have more aesthetically pleasing advantages over normal cartography projections such as Hugin's Panini projection - named after Italian vedutismo
painter Giovanni Paolo Pannini
- or PTgui's Vedutismo projection. Different projections may be combined in same image for fine tuning the final look of the output image.
Stereographic projection
or fisheye
projection can be used to form a little planet panorama by pointing the virtual camera straight down and setting the field of view
large enough to show the whole ground and some of the areas above it; pointing the virtual camera upwards creates a tunnel effect. Conformality of the stereographic projection may produce more visually pleasing result than equal area fisheye projection as discussed in the stereographic projection's article.
of the camera) can lead to parallax
errors in the final product. When the captured scene features rapid movement or dynamic motion, artifacts may occur as a result of time differences between the image segments. "Blind stitching" through feature-based alignment methods (see autostitch
), as opposed to manual selection and stitching, can cause imperfections in the assembly of the panorama.
, Hugin
, Ptgui
, Panorama Tools, Microsoft Research Image Composite Editor
, and CleVR Stitcher
. Many other programs can also stitch multiple images; a popular example is Adobe Systems
' Photoshop, which includes a tool known as Photomerge and, in the latest versions, the new Auto-Blend.
Photograph
A photograph is an image created by light falling on a light-sensitive surface, usually photographic film or an electronic imager such as a CCD or a CMOS chip. Most photographs are created using a camera, which uses a lens to focus the scene's visible wavelengths of light into a reproduction of...
ic image
Image
An image is an artifact, for example a two-dimensional picture, that has a similar appearance to some subject—usually a physical object or a person.-Characteristics:...
s with overlapping fields of view to produce a segmented panorama
Panorama
A panorama is any wide-angle view or representation of a physical space, whether in painting, drawing, photography, film/video, or a three-dimensional model....
or high-resolution image. Commonly performed through the use of computer software
Computer software
Computer software, or just software, is a collection of computer programs and related data that provide the instructions for telling a computer what to do and how to do it....
, most approaches to image stitching require nearly exact overlaps between images and identical exposures to produce seamless results. Some digital camera
Digital camera
A digital camera is a camera that takes video or still photographs, or both, digitally by recording images via an electronic image sensor. It is the main device used in the field of digital photography...
s can stitch their photos internally.
Stages of the stitching process
Image registration involves matching features in a set of images or using direct alignment methods to search for image alignments that minimize the sum of absolute differences
Sum of absolute differences
Sum of absolute differences is a widely used, extremely simple algorithm for measuring the similarity between image blocks. It works by taking the absolute difference between each pixel in the original block and the corresponding pixel in the block being used for comparison...
between overlapping pixels. When using direct alignment methods one might first calibrate one's images to get better results. Additionally, users may input a rough model of the panorama to help the feature matching stage, so that - for example - only neighboring images are searched for matching features. Since there are smaller group of features for matching, the result of the search is more accurate and execution of the comparison is faster.
Image calibration aims to minimize differences between an ideal lens models and the camera-lens combination that was used, optical defects such as distortions, exposure
Exposure (photography)
In photography, exposure is the total amount of light allowed to fall on the photographic medium during the process of taking a photograph. Exposure is measured in lux seconds, and can be computed from exposure value and scene luminance over a specified area.In photographic jargon, an exposure...
differences between images, vignetting
Vignetting
In photography and optics, vignetting is a reduction of an image's brightness or saturation at the periphery compared to the image center. The word vignette, from the same root as vine, originally referred to a decorative border in a book. Later, the word came to be used for a photographic...
, camera response and chromatic aberrations
Chromatic aberration
In optics, chromatic aberration is a type of distortion in which there is a failure of a lens to focus all colors to the same convergence point. It occurs because lenses have a different refractive index for different wavelengths of light...
. If feature detection methods were used to register images and absolute positions of the features were recorded and saved, stitching software may use the data for geometric optimization of the images in addition to placing the images on the panosphere. Panotools and its various derivative programs use this method.
Image blending involves executing the adjustments figured out in the calibration stage, combined with remapping of the images to an output projection. Colors are adjusted
Color mapping
Color mapping is a function that maps the colors of one image to the colors of another image. A color mapping may be referred to as the algorithm that results in the mapping function or the algorithm that transforms the image colors...
between images to compensate for exposure differences. If applicable, high dynamic range
High dynamic range imaging
In image processing, computer graphics, and photography, high dynamic range imaging is a set of techniques that allows a greater dynamic range between the lightest and darkest areas of an image than current standard digital imaging techniques or photographic methods...
merging is done along with motion compensation
Image registration
Image registration is the process of transforming different sets of data into one coordinate system. Data may be multiple photographs, data from different sensors, from different times, or from different viewpoints. It is used in computer vision, medical imaging, military automatic target...
and deghosting. Images are blended together and seam line adjustment is done to minimize the visibility of seams between images.
Projection geometry
For image segments that have been taken from the same point in space, stitched images can be arranged using one of various map projectionMap projection
A map projection is any method of representing the surface of a sphere or other three-dimensional body on a plane. Map projections are necessary for creating maps. All map projections distort the surface in some fashion...
s.
Rectilinear projection, where the stitched image is viewed on a two-dimensional plane intersecting the panosphere in a single point. Lines that are straight in reality are shown as straight regardless of their directions on the image. Wide views - around 120° or so - start to exhibit severe distortion near the image borders. One case of rectilinear projection is the use of cube faces with cubic mapping
Cube mapping
In computer graphics, cube mapping is a method of environment mapping that uses a six-sided cube as the map shape. The environment is projected onto the six faces of a cube and stored as six square textures, or unfolded into six regions of a single texture...
for panorama viewing. Panorama is mapped to six squares, each cube face showing 90 by 90 degree area of the panorama.
Cylindrical projection, where the stitched image shows a 360° horizontal field of view and a limited vertical field of view. Panoramas in this projection are meant to be viewed as though the image is wrapped into a cylinder and viewed from within. When viewed on a 2D plane, horizontal lines appear curved while vertical lines remain straight. Vertical distortion increases rapidly when nearing the top of the panosphere. There are various other cylindrical formats, such as Mercator
Mercator projection
The Mercator projection is a cylindrical map projection presented by the Belgian geographer and cartographer Gerardus Mercator, in 1569. It became the standard map projection for nautical purposes because of its ability to represent lines of constant course, known as rhumb lines or loxodromes, as...
and Miller cylindrical which have less distortion near the poles of the panosphere.
Spherical projection or equirectangular projection
Equirectangular projection
The equirectangular projection is a very simple map projection attributed to Marinus of Tyre, who Ptolemy claims invented the projection about AD 100...
— which is strictly speaking another cylindrical projection — where the stitched image shows a 360° horizontal by 180° vertical field of view i.e the whole sphere. Panoramas in this projection are meant to be viewed as though the image is wrapped into a sphere and viewed from within. When viewed on a 2D plane, horizontal lines appear curved as in a cylindrical projection, while vertical lines remain vertical.
Since a panorama is basically map of a sphere, various other mapping projections from cartographers
Cartography
Cartography is the study and practice of making maps. Combining science, aesthetics, and technique, cartography builds on the premise that reality can be modeled in ways that communicate spatial information effectively.The fundamental problems of traditional cartography are to:*Set the map's...
can also be used if so desired. Additionally there are specialized projections which may have more aesthetically pleasing advantages over normal cartography projections such as Hugin's Panini projection - named after Italian vedutismo
Veduta
A veduta is a highly detailed, usually large-scale painting of a cityscape or some other vista....
painter Giovanni Paolo Pannini
Giovanni Paolo Pannini
Giovanni Paolo Panini or Pannini was a painter and architect, who worked in Rome and is mainly known as one of the vedutisti ....
- or PTgui's Vedutismo projection. Different projections may be combined in same image for fine tuning the final look of the output image.
Stereographic projection
Stereographic projection
The stereographic projection, in geometry, is a particular mapping that projects a sphere onto a plane. The projection is defined on the entire sphere, except at one point — the projection point. Where it is defined, the mapping is smooth and bijective. It is conformal, meaning that it...
or fisheye
Fisheye
Fisheye may refer to:* The eye of a fish or other aquatic creature resembling a fish* Fisheye lens, used in photography* Fisheye, a character from the anime Sailor Moon* FishEye , a revision control browser by Atlassian Software Systems...
projection can be used to form a little planet panorama by pointing the virtual camera straight down and setting the field of view
Field of view
The field of view is the extent of the observable world that is seen at any given moment....
large enough to show the whole ground and some of the areas above it; pointing the virtual camera upwards creates a tunnel effect. Conformality of the stereographic projection may produce more visually pleasing result than equal area fisheye projection as discussed in the stereographic projection's article.
Challenges of image stitching
The use of images not taken from the same place (on a pivot about the entrance pupilEntrance pupil
In an optical system, the entrance pupil is the optical image of the physical aperture stop, as 'seen' through the front of the lens system. The corresponding image of the aperture as seen through the back of the lens system is called the exit pupil...
of the camera) can lead to parallax
Parallax
Parallax is a displacement or difference in the apparent position of an object viewed along two different lines of sight, and is measured by the angle or semi-angle of inclination between those two lines. The term is derived from the Greek παράλλαξις , meaning "alteration"...
errors in the final product. When the captured scene features rapid movement or dynamic motion, artifacts may occur as a result of time differences between the image segments. "Blind stitching" through feature-based alignment methods (see autostitch
Autostitch
AutoStitch is a proprietary image stitching software tool for creating panoramas. It was developed by Matthew Brown and David G. Lowe of the University of British Columbia.The software uses methods known as SIFT and RANSAC...
), as opposed to manual selection and stitching, can cause imperfections in the assembly of the panorama.
Image stitching software
Dedicated programs include AutostitchAutostitch
AutoStitch is a proprietary image stitching software tool for creating panoramas. It was developed by Matthew Brown and David G. Lowe of the University of British Columbia.The software uses methods known as SIFT and RANSAC...
, Hugin
Hugin (software)
Hugin is a cross-platform open source panorama photo stitching and HDR merging program developed by Pablo d'Angelo and others. It is a GUI front-end for Helmut Dersch's Panorama Tools and Andrew Mihal's Enblend and Enfuse...
, Ptgui
PTgui
PTgui is a commercial panorama photo stitching program for Windows and Mac OS X developed by New House Internet Services BV. PTgui was initially created as a GUI frontend to Helmut Dersch's Panorama Tools. It currently features its own stitching and blending engine along with compatibility to...
, Panorama Tools, Microsoft Research Image Composite Editor
Microsoft Research Image Composite Editor
Microsoft Image Composite Editor is an advanced panoramic image stitcher made by the Microsoft Research division of Microsoft Corporation.The application takes a set of overlapping photographs of a scene shot from a single camera location and creates a high-resolution panorama incorporating all the...
, and CleVR Stitcher
CleVR
CleVR is a free panoramic photo sharing site and photo stitching software. It allows panoramas to be embedded into other web pages using a Flash viewer. Panoramas can be displayed with hotspots — areas in the scene that can be clicked to display other content or to navigate to another scene...
. Many other programs can also stitch multiple images; a popular example is Adobe Systems
Adobe Systems
Adobe Systems Incorporated is an American computer software company founded in 1982 and headquartered in San Jose, California, United States...
' Photoshop, which includes a tool known as Photomerge and, in the latest versions, the new Auto-Blend.