Free viewpoint television
Encyclopedia
Free viewpoint television (FTV) is a system for viewing natural video, allowing the user to interactively control the viewpoint and generate new views of a dynamic scene from any 3D position.
The equivalent system for computer-simulated
video is known as virtual reality
. With FTV, the focus of attention can be controlled by the viewers rather than a director, meaning that each viewer may be observing a unique viewpoint. It remains to be seen how FTV will affect television watching as a group activity.
community for a long time but only in recent years has the speed and quality reached levels that are suitable for serious consideration as an end user system.
Professor Masayuki Tanimoto from Nagoya University
(Japan) has done much to promote the use of the term "free viewpoint television" and has published many papers on the ray space representation, although other techniques can be, and are used for FTV.
QuickTime VR
might be considered a predecessor to FTV.
It is not enough to simply place cameras around the scene to be captured. The geometry of the camera set up must be measured by a process known in computer vision as "camera calibration." Manual alignment would be too cumbersome so typically a "best effort" alignment is performed prior to capturing a test pattern that is used to generate calibration parameters.
Restricted free viewpoint television views for large environments can be captured from a single location camera system mounted on a moving platform. Depth data must also be captured, which is necessary to generate the free viewpoint. The Google Street View
capture system is an example with limited functionality. The first full commercial implementation, iFlex, was delivered in 2009 by Real Time Race
.
Multiview video capture varies from partial (usually about 30 degrees) to complete (360 degrees) coverage of the scene. Therefore, it is possible to output stereoscopic views suitable for viewing with a 3D display or other 3D methods. Systems with more physical cameras can capture images with more coverage of the viewable scene, however, it is likely that certain regions will always be occluded from any viewpoint. A larger number of cameras should make it possible to obtain high quality output because less interpolation is needed.
More cameras mean that efficient coding of the Multiview Video is required. This may not be such a big disadvantage as there are representations that can remove the redundancy in MVV; such as inter view coding using MPEG-4
or Multiview Video Coding
, the ray space representation, geometry videos, etc.
In terms of hardware, the user requires a viewing device that can decode MVV and synthesize new viewpoints, and a 2D or 3D display.
(MPEG) has normalized Annex H of MPEG-4 AVC
in March 2009 call Multiview Video Coding
after the work of a group called '3DAV' (3D Audio and Visual) headed by Aljoscha Smolic at the Heinrich-Hertz Institute.
The equivalent system for computer-simulated
Computer simulation
A computer simulation, a computer model, or a computational model is a computer program, or network of computers, that attempts to simulate an abstract model of a particular system...
video is known as virtual reality
Virtual reality
Virtual reality , also known as virtuality, is a term that applies to computer-simulated environments that can simulate physical presence in places in the real world, as well as in imaginary worlds...
. With FTV, the focus of attention can be controlled by the viewers rather than a director, meaning that each viewer may be observing a unique viewpoint. It remains to be seen how FTV will affect television watching as a group activity.
History
Systems for rendering arbitrary views of natural scenes have been well known in the computer visionComputer vision
Computer vision is a field that includes methods for acquiring, processing, analysing, and understanding images and, in general, high-dimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions...
community for a long time but only in recent years has the speed and quality reached levels that are suitable for serious consideration as an end user system.
Professor Masayuki Tanimoto from Nagoya University
Nagoya University
Nagoya University is one of the most prestigious universities in Japan. It can be seen in the several rankings such as shown below.-General Rankings:...
(Japan) has done much to promote the use of the term "free viewpoint television" and has published many papers on the ray space representation, although other techniques can be, and are used for FTV.
QuickTime VR
QuickTime VR
QuickTime VR is a type of image file format developed by Apple Inc. for QuickTime. It allows the creation and viewing of photographically-captured panoramas and the exploration of objects through images taken at multiple viewing angles...
might be considered a predecessor to FTV.
Capture and display
In order to acquire the views necessary to allow a high quality rendering of the scene from any angle, several cameras are placed around the scene; either in a studio environment or an outdoor venue, such as a sporting arena for example. The output Multiview Video (MVV) must then be packaged suitably so that the data may be compressed and also so that the users' viewing device may easily access the relevant views to interpolate new views.It is not enough to simply place cameras around the scene to be captured. The geometry of the camera set up must be measured by a process known in computer vision as "camera calibration." Manual alignment would be too cumbersome so typically a "best effort" alignment is performed prior to capturing a test pattern that is used to generate calibration parameters.
Restricted free viewpoint television views for large environments can be captured from a single location camera system mounted on a moving platform. Depth data must also be captured, which is necessary to generate the free viewpoint. The Google Street View
Google Street View
Google Street View is a technology featured in Google Maps and Google Earth that provides panoramic views from various positions along many streets in the world...
capture system is an example with limited functionality. The first full commercial implementation, iFlex, was delivered in 2009 by Real Time Race
Real Time Race
Real Time Race Limited is a company based at Daresbury Laboratory in North West England. The company was Founded in 2003 by Christopher Leigh and Stuart Scott-Goldstone....
.
Multiview video capture varies from partial (usually about 30 degrees) to complete (360 degrees) coverage of the scene. Therefore, it is possible to output stereoscopic views suitable for viewing with a 3D display or other 3D methods. Systems with more physical cameras can capture images with more coverage of the viewable scene, however, it is likely that certain regions will always be occluded from any viewpoint. A larger number of cameras should make it possible to obtain high quality output because less interpolation is needed.
More cameras mean that efficient coding of the Multiview Video is required. This may not be such a big disadvantage as there are representations that can remove the redundancy in MVV; such as inter view coding using MPEG-4
MPEG-4
MPEG-4 is a method of defining compression of audio and visual digital data. It was introduced in late 1998 and designated a standard for a group of audio and video coding formats and related technology agreed upon by the ISO/IEC Moving Picture Experts Group under the formal standard ISO/IEC...
or Multiview Video Coding
Multiview Video Coding
Multiview Video Coding is an amendment to H.264/MPEG-4 AVC video compression standard developed with joint efforts by MPEG/VCEG that enables efficient encoding of sequences captured simultaneously from multiple cameras using a single video stream....
, the ray space representation, geometry videos, etc.
In terms of hardware, the user requires a viewing device that can decode MVV and synthesize new viewpoints, and a 2D or 3D display.
Standardization
The Moving Picture Experts GroupMoving Picture Experts Group
The Moving Picture Experts Group is a working group of experts that was formed by ISO and IEC to set standards for audio and video compression and transmission. It was established in 1988 by the initiative of Hiroshi Yasuda and Leonardo Chiariglione, who has been from the beginning the Chairman...
(MPEG) has normalized Annex H of MPEG-4 AVC
H.264/MPEG-4 AVC
H.264/MPEG-4 Part 10 or AVC is a standard for video compression, and is currently one of the most commonly used formats for the recording, compression, and distribution of high definition video...
in March 2009 call Multiview Video Coding
Multiview Video Coding
Multiview Video Coding is an amendment to H.264/MPEG-4 AVC video compression standard developed with joint efforts by MPEG/VCEG that enables efficient encoding of sequences captured simultaneously from multiple cameras using a single video stream....
after the work of a group called '3DAV' (3D Audio and Visual) headed by Aljoscha Smolic at the Heinrich-Hertz Institute.
External links
- iview is a British DTI project between BBCBBCThe British Broadcasting Corporation is a British public service broadcaster. Its headquarters is at Broadcasting House in the City of Westminster, London. It is the largest broadcaster in the world, with about 23,000 staff...
, Snell & WilcoxSnell & WilcoxSnell Limited is a company that designs and develops solutions for the digital media market including applications for central operations, live production, post production, playout and media management...
and University of SurreyUniversity of SurreyThe University of Surrey is a university located within the county town of Guildford, Surrey in the South East of England. It received its charter on 9 September 1966, and was previously situated near Battersea Park in south-west London. The institution was known as Battersea College of Technology...
to develop an FTV system. - Eye Vision is a system developed by Professor Takeo KanadeTakeo Kanadeis a Japanese computer scientist and one of the world's foremost researchers in computer vision. He is currently U.A. and Helen Whitaker Professor at Carnegie Mellon University...
at CMU for CBS's coverage of Super Bowl XXXVSuper Bowl XXXVSuper Bowl XXXV was played on January 28, 2001 at Raymond James Stadium in Tampa, Florida to decide the National Football League champion following the 2000 regular season. The American Football Conference champion Baltimore Ravens defeated the National Football Conference champion New York...
. The user is not able to change viewpoint but the camera operator is able to choose any virtual viewpoint by synthesizing images from an active visionActive visionAn area of computer vision is active vision, sometimes also called active computer vision. An active vision system is one that can manipulate the viewpoint of the camera in order to investigate the environment and get better information from it...
system.