Time-of-flight camera
Encyclopedia
A time-of-flight camera (TOF camera) is a range imaging
camera system that resolves distance based on the known speed of light
, measuring the time-of-flight
of a light signal between the camera and the subject. The time-of-flight camera is a class of flash-imaging LIDAR
, in which the entire scene is captured with each laser or light pulse, rather than being scanned point-by-point with a moving laser as in scanning.
Time-of-flight cameras are relatively recent devices for normal products (2004), as the semiconductor processes have only recently become fast enough for such devices. The systems cover ranges of a few meters up to about several kilometers depending upon the detector material being used.
The detector material is made from (1) CMOS detectors which capture light pulses in the visible range, (2) PIN diodes or (3) avalanche photo diodes (APDs). Each material has inherent strengths and are chosen based upon application requirements.
The distance resolution ranges from sub-centimeter to several centimeters depending upon the range. The lateral resolution of time-of-flight cameras is generally low compared to standard 2D video cameras, with most commercially available devices at 320 × 240 pixels or less as of 2011. Unlike 3D laser scanning
methods of capturing 3D images, TOF cameras can operate very quickly, providing up to 100 images per second.
was purchased by Microsoft in 2010.
The distance can be calculated using the equation, z = R (S2 − S1) / 2(S1 + S2) + R / 2 for an ideal camera. R is the camera range, determined by the round trip of the light pulse, S1 the amount of the light pulse that is received, and S2 the amount of the light pulse that is blocked.
The ZCam
by 3DV Systems is a range-gated system. Microsoft purchased 3DV in 2009.
Similar principles are used in the ToF camera line developed by the Fraunhofer
Institute of Microelectronic Circuits and Systems and TriDiCam. These cameras employ photodetectors with a fast electronic shutter.
Range gated imagers can also be used in 2D imaging to suppress anything outside a specified distance range, such as to see through fog. A pulsed laser provides illumination, and an optical gate allows light to reach the imager only during the desired time period.
The pulse width of the illumination determines the maximum range the camera can handle. With a pulse width of e.g. 50 ns, the range is limited to
These short times show that the illumination unit is a critical part of the system. Only with some special LEDs or lasers is it possible to generate such short pulses.
The single pixel consists of a photo sensitive element (e.g. a photo diode). It converts the incoming light into a current. In analog timing imagers, connected to the photo diode are fast switches, which direct the current to one of two (or several) memory elements (e.g. a capacitor
) that act as summation elements. In digital timing imagers, a time counter, running at several gigahertz, is connected to each photodetector pixel and stops counting when light is sensed.
In the diagram of an analog timer, the pixel uses two switches (G1 and G2) and two memory elements (S1 and S2). The switches are controlled by a pulse with the same length as the light pulse, where the control signal of switch G2 is delayed by exactly the pulse width. Depending on the delay, only part of the light pulse is sampled through G1 in S1, the other part is stored in S2. Depending on the distance, the ratio between S1 and S2 changes as depicted in the drawing. Because only small amounts of light hit the sensor within 50 ns, not only one but several thousands pulses are sent out (repetition rate tR) and gathered, thus increasing the signal to noise ratio.
After the exposure, the pixel is read out and the following stages measure the signals S1 and S2. As the length of the light pulse is defined, the distance can be calculated with Medina's formula:
In the example, the signals have the following values: S1 = 0.66 and S2 = 0.33. The distance is therefore:
In the presence of background light, the memory elements receive an additional part of the signal. This would disturb the distance measurement. To eliminate the background part of the signal, the whole measurement can be performed a second time with the illumination switched off. If the objects are further away than the distance range, the result is also wrong. Here, a second measurement with the control signals delayed by an additional pulse width helps to suppress such objects.
Other systems work with a sinusoidally modulated light source instead of the pulse source.
or triangulation systems
, the whole system is very compact: the illumination is placed just next to the lens, whereas the other systems need a certain minimum base line. In contrast to laser scanning systems
, no mechanical moving parts are needed.
After the distance data has been extracted, object detection, for example, is also easy to carry out because the algorithms are not disturbed by patterns on the object.
Other applications are measurement tasks, e.g. for the fill height in silos. In industrial machine vision, the time-of-flight camera helps to classify objects and help robots find the items, for instance on a conveyor. Door controls can distinguish easily between animals and humans reaching the door.
Defunct brands
Range imaging
Range imaging is the name for a collection of techniques which are used to produce a 2D image showing the distance to points in a scene from a specific point, normally associated with some type of sensor device....
camera system that resolves distance based on the known speed of light
Speed of light
The speed of light in vacuum, usually denoted by c, is a physical constant important in many areas of physics. Its value is 299,792,458 metres per second, a figure that is exact since the length of the metre is defined from this constant and the international standard for time...
, measuring the time-of-flight
Time-of-flight
Time of flight describes a variety of methods that measure the time that it takes for an object, particle or acoustic, electromagnetic or other wave to travel a distance through a medium...
of a light signal between the camera and the subject. The time-of-flight camera is a class of flash-imaging LIDAR
3D Flash LIDAR
- About LIDAR :LIDAR, is an acronym which stands for Light Detection And Ranging based on a laser-radar paradigm. RADAR is the process of transmitting, receiving, detecting, and processing and electromagnetic wave that reflects from a target, a technology that was developed by the German Army in...
, in which the entire scene is captured with each laser or light pulse, rather than being scanned point-by-point with a moving laser as in scanning.
Time-of-flight cameras are relatively recent devices for normal products (2004), as the semiconductor processes have only recently become fast enough for such devices. The systems cover ranges of a few meters up to about several kilometers depending upon the detector material being used.
The detector material is made from (1) CMOS detectors which capture light pulses in the visible range, (2) PIN diodes or (3) avalanche photo diodes (APDs). Each material has inherent strengths and are chosen based upon application requirements.
The distance resolution ranges from sub-centimeter to several centimeters depending upon the range. The lateral resolution of time-of-flight cameras is generally low compared to standard 2D video cameras, with most commercially available devices at 320 × 240 pixels or less as of 2011. Unlike 3D laser scanning
3D scanner
A 3D scanner is a device that analyzes a real-world object or environment to collect data on its shape and possibly its appearance . The collected data can then be used to construct digital, three dimensional models....
methods of capturing 3D images, TOF cameras can operate very quickly, providing up to 100 images per second.
Pulsed light source with digital time counters
There are devices with a pulsed laser and a custom imaging integrated circuit with a fast counter behind every pixel. These devices produce depth values for each pixel on every frame. Typical image sizes are 128 x 128 pixels. Ranges up to 22,000 feet with an eye-safe narrow beam have been achieved. Detectors are typically InGaAs (indium-gallium-arsenide) devices.RF-modulated light sources with phase detectors
Photonic Mixer Devices (PMD), the Swiss Ranger, and CanestaVision work by modulating the outgoing beam with an RF carrier, then measuring the phase shift of that carrier on the receive side. This approach has a modular error challenge; ranges are mod the maximum range, which is the RF carrier wavelength. The Swiss Ranger is a compact, short-range device, with ranges of 5 or 10 meters, with 176 x 144 pixels. With phase unwrapping algorithms, the maximum uniqueness range can be increased. The PMD can provide ranges up to 60m. Illumination is pulsed LEDs, rather than a laser. CanestaVision developer CanestaCanesta
Canesta is a fabless semiconductor company founded in April 1999 by Cyrus Bamji, Abbas Rafii, and Nazim Kareemi. The company manufactures CMOS-based single chip 3D sensors, which can be used as part of input systems for electronic devices. On October 29, 2010, Canesta announced that it would be...
was purchased by Microsoft in 2010.
Range gated imagers
These devices have a built-in shutter in front of the image sensor that opens and closes at the same rate as the light pulses are sent out. Because part of every returning pulse is blocked by the shutter according to its time of arrival, the amount of light received relates to the distance the pulse has traveled.The distance can be calculated using the equation, z = R (S2 − S1) / 2(S1 + S2) + R / 2 for an ideal camera. R is the camera range, determined by the round trip of the light pulse, S1 the amount of the light pulse that is received, and S2 the amount of the light pulse that is blocked.
The ZCam
ZCam
ZCam is a brand of time-of-flight camera products for video applications by Israeli developer 3DV Systems. The ZCam supplements full-color video camera imaging with real-time range imaging information, allowing for the capture of video in 3D....
by 3DV Systems is a range-gated system. Microsoft purchased 3DV in 2009.
Similar principles are used in the ToF camera line developed by the Fraunhofer
Fraunhofer Society
The Fraunhofer Society is a German research organization with 60 institutes spread throughout Germany, each focusing on different fields of applied science . It employs around 18,000, mainly scientists and engineers, with an annual research budget of about €1.65 billion...
Institute of Microelectronic Circuits and Systems and TriDiCam. These cameras employ photodetectors with a fast electronic shutter.
Range gated imagers can also be used in 2D imaging to suppress anything outside a specified distance range, such as to see through fog. A pulsed laser provides illumination, and an optical gate allows light to reach the imager only during the desired time period.
Components
A time-of-flight camera consists of the following components:- Illumination unit: It illuminates the scene. As the light has to be modulated with high speeds up to 100 MHz, only LEDLEdLEd is a TeX/LaTeX editing software working under Microsoft Windows. It is a freeware product....
s or laser diodeLaser diodeThe laser diode is a laser where the active medium is a semiconductor similar to that found in a light-emitting diode. The most common type of laser diode is formed from a p-n junction and powered by injected electric current...
s are feasible. The illumination normally uses infrared light to make the illumination unobtrusive. - Optics: A lens gathers the reflected light and images the environment onto the image sensor. An optical band pass filter only passes the light with the same wavelength as the illumination unit. This helps suppress background light.
- Image sensor: This is the heart of the TOF camera. Each pixel measures the time the light has taken to travel from the illumination unit to the object and back. Several different approaches are used for timing; see types of devices above.
- Driver electronics: Both the illumination unit and the image sensor have to be controlled by high speed signals. These signals have to be very accurate to obtain a high resolution. For example, if the signals between the illumination unit and the sensor shift by only 10 picosecondPicosecondA picosecond is 10−12 of a second. That is one trillionth, or one millionth of one millionth of a second, or 0.000 000 000 001 seconds. A picosecond is to one second as one second is to 31,700 years....
s, the distance changes by 1.5 mm. For comparison: current CPUs reach frequencies of up to 3 GHzGHZGHZ or GHz may refer to:# Gigahertz .# Greenberger-Horne-Zeilinger state — a quantum entanglement of three particles.# Galactic Habitable Zone — the region of a galaxy that is favorable to the formation of life....
, corresponding to clock cycles of about 300 ps - the corresponding 'resolution' is only 45 mm. - Computation/Interface: The distance is calculated directly in the camera. To obtain good performance, some calibration data is also used. The camera then provides a distance image over a USB or EthernetEthernetEthernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
interface.
Principle
The simplest version of a time-of-flight camera uses light pulses. The illumination is switched on for a very short time, the resulting light pulse illuminates the scene and is reflected by the objects. The camera lens gathers the reflected light and images it onto the sensor plane. Depending on the distance, the incoming light experiences a delay. As light has a speed of approximately c = 300,000,000 meters per second, this delay is very short: an object 2.5 m away will delay the light by:The pulse width of the illumination determines the maximum range the camera can handle. With a pulse width of e.g. 50 ns, the range is limited to
These short times show that the illumination unit is a critical part of the system. Only with some special LEDs or lasers is it possible to generate such short pulses.
The single pixel consists of a photo sensitive element (e.g. a photo diode). It converts the incoming light into a current. In analog timing imagers, connected to the photo diode are fast switches, which direct the current to one of two (or several) memory elements (e.g. a capacitor
Capacitor
A capacitor is a passive two-terminal electrical component used to store energy in an electric field. The forms of practical capacitors vary widely, but all contain at least two electrical conductors separated by a dielectric ; for example, one common construction consists of metal foils separated...
) that act as summation elements. In digital timing imagers, a time counter, running at several gigahertz, is connected to each photodetector pixel and stops counting when light is sensed.
In the diagram of an analog timer, the pixel uses two switches (G1 and G2) and two memory elements (S1 and S2). The switches are controlled by a pulse with the same length as the light pulse, where the control signal of switch G2 is delayed by exactly the pulse width. Depending on the delay, only part of the light pulse is sampled through G1 in S1, the other part is stored in S2. Depending on the distance, the ratio between S1 and S2 changes as depicted in the drawing. Because only small amounts of light hit the sensor within 50 ns, not only one but several thousands pulses are sent out (repetition rate tR) and gathered, thus increasing the signal to noise ratio.
After the exposure, the pixel is read out and the following stages measure the signals S1 and S2. As the length of the light pulse is defined, the distance can be calculated with Medina's formula:
In the example, the signals have the following values: S1 = 0.66 and S2 = 0.33. The distance is therefore:
In the presence of background light, the memory elements receive an additional part of the signal. This would disturb the distance measurement. To eliminate the background part of the signal, the whole measurement can be performed a second time with the illumination switched off. If the objects are further away than the distance range, the result is also wrong. Here, a second measurement with the control signals delayed by an additional pulse width helps to suppress such objects.
Other systems work with a sinusoidally modulated light source instead of the pulse source.
Simplicity
In contrast to stereo visionStereo camera
A stereo camera is a type of camera with two or more lenses with a separate image sensor or film frame for each lens. This allows the camera to simulate human binocular vision, and therefore gives it the ability to capture three-dimensional images, a process known as stereo photography. Stereo...
or triangulation systems
Range imaging
Range imaging is the name for a collection of techniques which are used to produce a 2D image showing the distance to points in a scene from a specific point, normally associated with some type of sensor device....
, the whole system is very compact: the illumination is placed just next to the lens, whereas the other systems need a certain minimum base line. In contrast to laser scanning systems
3D scanner
A 3D scanner is a device that analyzes a real-world object or environment to collect data on its shape and possibly its appearance . The collected data can then be used to construct digital, three dimensional models....
, no mechanical moving parts are needed.
Efficient distance algorithm
It is very easy to extract the distance information out of the output signals of the TOF sensor, therefore this task uses only a small amount of processing power, again in contrast to stereo vision, where complex correlation algorithms have to be implemented.After the distance data has been extracted, object detection, for example, is also easy to carry out because the algorithms are not disturbed by patterns on the object.
Speed
Time-of-flight cameras are able to measure the distances within a complete scene with one shot. As the cameras reach up to 100 frames per second, they are ideally suited to be used in real-time applications.Background light
Although most of the background light coming from artificial lighting or the sun is suppressed, the pixel still has to provide a high dynamic range. The background light also generates electrons, which have to be stored. For example, the illumination units in today's TOF cameras can provide an illumination level of about 1 watt. The Sun has an illumination power of about 50 watts per square meter after the optical bandpass filter. Therefore, if the illuminated scene has a size of 1 square meter, the light from the sun is 50 times stronger than the modulated signal.Interference
If several time-of-flight cameras are running at the same time, the cameras may disturb each others' measurements. There exist several possibilities for dealing with this problem:- Time multiplexing: A control system starts the measurement of the individual cameras consecutively, so that only one illumination unit is active at a time.
- Different modulation frequencies: If the cameras modulate their light with different modulation frequencies, their light is collected in the other systems only as background illumination but does not disturb the distance measurement.
Multiple reflections
In contrast to laser scanning systems, where only a single point is illuminated at once, the time-of-flight cameras illuminate a whole scene. Due to multiple reflections, the light may reach the objects along several paths and therefore, the measured distance may be greater than the true distance.Applications
Automotive applications
Time-of-flight cameras are also used in assistance and safety functions for advanced automotive applications such as active pedestrian safety, precrash detection and indoor applications like out-of-position (OOP) detection.Human-machine interfaces and gaming
As time-of-flight cameras provide distance images in real time, it is easy to track movements of humans. This allows new interactions with consumer devices such as televisions. Another topic is to use this type of cameras to interact with games on video game consoles.Measurement and machine vision
Robotics
Another use of these cameras is the field of robotics: Mobile robots can build up a map of their surroundings very quickly, enabling them to avoid obstacles or follow a leading person. As the distance calculation is simple, only little computational power is used.Brands
Active brands (as of 2011)- D-IMager - TOF camera by Panasonic Electric Works
- DepthSense - TOF cameras and modules by SoftKineticSoftkineticSoftKinetic is a Belgian company which develops gesture recognition hardware and software for real-time range imaging cameras . It was founded in July 2007. SoftKinetic provides gesture recognition solutions based on its technology to the interactive digital entertainment, consumer electronics,...
- FotonicFotonicFotonic is a company focused on the use of time-of-flight technology for three-dimensional imaging. The company has a special focus on developing time-of-flight cameras that meet the demands of modern industry in terms of accuracy, reliability and robustness.Fotonic’s customers develop...
- TOF cameras and software powered by Canesta CMOS chip - PMD[vision] - TOF imager, modules, cameras and software by PMDTechnologiesPMDTechnologiesPMDTechnologies is a developer of CMOS semiconductor 3D time-of-flight components and a provider of engineering support in the field of digital 3D imaging. The company is named after the Photonic Mixer Device technology used in its products to detect 3D data in real time...
- SwissRanger - an industrial TOF-only camera line originally by the Centre Suisse d'Electronique et Microtechnique, S.A.S.A. (corporation)S.A. designates a particular type of corporation in various countries, mostly those employing the civil law. It originated in Spain during the 16th century. Depending on language, the abbreviation stands for various phrases meaning anonymous society, anonymous company, anonymous partnership, or...
(CSEMSwiss Center for Electronics and MicrotechnologyCSEM SA is a Swiss Research and Development company with expertise in Micro Nano Technologies, Microelectronics, Systems Engineering und Communication Technologies. The headquarter is situated in Neuchâtel. CSEM also has centers in the cities Muttenz, Zürich, Alpnach and Landquart in Switzerland...
), now developed by the spin outSpin outA spin-out, also known as a spin-off or a starburst, refers to a type of corporate action where a company "splits off" sections of itself as a separate business....
company Mesa ImagingMESA ImagingMESA Imaging is a time-of-flight camera company. A privately financed organization, MESA was founded in July 2006 as a spin out from the Swiss Center for Electronics and Microtechnology to commercialize its time-of-flight camera technologies... - 3D MLI Sensor - TOF imager, modules, cameras, and software by IEE (International Electronics & Engineering), based on modulated light intensity (MLI)
- 2+3D - High-resolution SXGA (1280×1024) TOF camera in development by startup companyStartup companyA startup company or startup is a company with a limited operating history. These companies, generally newly created, are in a phase of development and research for markets...
odos imaging, integrating conventional image capture with TOF ranging in the same sensor. Based on technology developed at SiemensSiemensSiemens may refer toSiemens, a German family name carried by generations of telecommunications industrialists, including:* Werner von Siemens , inventor, founder of Siemens AG...
. - TOFCam Stanley - TOF camera by Stanley Electric
- TriDiCam - TOF modules and software, the TOF imager originally developed by FraunhoferFraunhofer SocietyThe Fraunhofer Society is a German research organization with 60 institutes spread throughout Germany, each focusing on different fields of applied science . It employs around 18,000, mainly scientists and engineers, with an annual research budget of about €1.65 billion...
Institute of Microelectronic Circuits and Systems, now developed by the spin out company TriDiCam
Defunct brands
- CanestaVision - TOF modules and software by CanestaCanestaCanesta is a fabless semiconductor company founded in April 1999 by Cyrus Bamji, Abbas Rafii, and Nazim Kareemi. The company manufactures CMOS-based single chip 3D sensors, which can be used as part of input systems for electronic devices. On October 29, 2010, Canesta announced that it would be...
(company acquired by Microsoft in 2010) - OptriCam - TOF cameras and modules by Optrima (rebranded DepthSense with SoftKinetic partnership in 2010)
- ZCamZCamZCam is a brand of time-of-flight camera products for video applications by Israeli developer 3DV Systems. The ZCam supplements full-color video camera imaging with real-time range imaging information, allowing for the capture of video in 3D....
- TOF camera products by 3DV Systems, integrating full-color video with depth information (assets sold to Microsoft in 2009)
External links
- ARTTS - Research project on time-of-flight cameras funded by the European Commission (under Information Society Technologies)
- Workshop on Time of Flight based Computer Vision (TOF-CV) at the 2008 IEEE Conference on Computer Vision and Pattern Recognition
- Gesturespace - a user interface design project based on time-of-flight cameras at the Zurich University of the Arts (ZHdK)
- "Calibration and Registration for Precise Surface Reconstruction with TOF Cameras" - Institute of Robotics and Mechatronics, German Aerospace CenterGerman Aerospace CenterThe German Aerospace Center is the national centre for aerospace, energy and transportation research of the Federal Republic of Germany. It has multiple locations throughout Germany. Its headquarters are located in Cologne. It is engaged in a wide range of research and development projects in...
- "First steps in enhancing 3D vision technique using 2D/3D sensors" - Center for Sensor Systems, University of SiegenUniversity of SiegenThe University of Siegen in Siegen, North Rhine-Westphalia, was founded in 1972. 14,100 students were enrolled at the university as of October 2010.-Faculties:University of Siegen offers in total 126 degree programmes across four faculties:...
- "Technological overview of Time-of-Flight cameras" - Description of the technology and comparision to other real-time 3-D acquisition techniques, Metrilus GmbH