Video for Virtual Reality

Christian Richardt¹    James Tompkin²    Jordan Halsey³    Aaron Hertzmann⁴    Jonathan Starck⁵    Oliver Wang⁴

¹ University of Bath       ² Brown University       ³ VR Playhouse       ⁴ Adobe Research       ⁵ Foundry

SIGGRAPH 2017 Course

Thursday 3 August 2017, 9:00–12:15, Los Angeles Convention Center, Room 403AB

Abstract

Video can capture the dynamic appearance of the real world in a way no other technology does; virtual reality technology, on the other hand, enables the display of dynamic visual content with unparalleled realism and immersion. The fusion of these two technologies—video for virtual reality (VR)—promises to enable many exciting photo-realistic experiences.

Over half a day, this course will provide an overview of three aspects of this exciting medium: the technical foundations, current systems in practice, and the potential for future systems of VR video.

In the first section, we will explore the geometric and optical problems underpinning VR video. Then, we will introduce both 360 degree video and stereoscopic video, including how 360 video is captured, analyzed, and stitched, including the mathematics behind how stereo 360 video can be captured.

This background material provides the prerequisites for understanding current systems in use. In the middle hour of the course, we explain how state-of-the-art stereo 360 video is produced from camera systems and computational processing. Then, we will consider the art of storytelling in VR, and how new tools for editing VR video can aid in the craft of this art production. Finally, this section provides an industry perspective covering current production and post-production choices and practice, including CG integration.

The final part of our course focuses on the next generation of video for VR, where we move to 6 degrees-of-freedom (6DoF) experiences. We introduce the basics and challenges behind light field cameras, processing and displays, and see how they can enable 6DoF experiences. This will be followed by another industry perspective on how light field camera arrays have been used to create cutting-edge experiences integrating volumetric live-action elements. To conclude the course, we will see how far we still must go toward the ideal system, in hopes of inspiring the attendees to push the boundary farther to reach it.

We hope this course is useful to a broad audience—at SIGGRAPH and beyond—as we cover the academic, artistic, and production sides of VR video.

@inproceedings{Video4VR,
  author    = {Christian Richardt and
               James Tompkin and
               Jordan Halsey and
               Aaron Hertzmann and
               Jonathan Starck and
               Oliver Wang},
  title     = {Video for Virtual Reality},
  booktitle = {SIGGRAPH Courses},
  year      = {2017},
  doi       = {10.1145/3084873.3084894},
  url       = {http://richardt.name/pub/Video4VR/},
}