Coherent Spatiotemporal Filtering,
Upsampling and Rendering of RGBZ Videos

Computer Graphics Forum (Proceedings of Eurographics 2012)

Abstract

Sophisticated video processing effects require both image and geometry information. We explore the possibility to augment a video camera with a recent infrared time-of-flight depth camera, to capture high-resolution RGB and low-resolution, noisy depth at video frame rates. To turn such a setup into a practical RGBZ video camera, we develop efficient data filtering techniques that are tailored to the noise characteristics of IR depth cameras. We first remove typical artefacts in the RGBZ data and then apply an efficient spatiotemporal denoising and upsampling scheme. This allows us to record temporally coherent RGBZ videos at interactive frame rates and to use them to render a variety of effects in unprecedented quality. We show effects such as video relighting, geometry-based abstraction and stylisation, background segmentation and rendering in stereoscopic 3D.

Version history

v1.1 (released 2012-09-17) — Windows binaries, source code, readme.txt

  • Fixed a locale bug that prevented floating-point values in config files from being read correctly if the machine's decimal separator is not a dot.
  • Fixed optical flow bug when FineLevel != 0.
  • BitmapFrames: added an implementation of fopen_s for non-Microsoft compilers.
  • Minor cosmetic code changes and cleanup.

v1.0 (released 2012-04-04) — Windows binaries, source code, readme.txt

  • Initial public version.

Bibtex

@article{RGBZcamera,
  author    = {Christian Richardt and Carsten Stoll and Neil A. Dodgson
                 and Hans-Peter Seidel and Christian Theobalt},
  title     = {Coherent Spatio\-temporal Filtering,
                 Upsampling and Rendering of {RGBZ} Videos},
  journal   = {Computer Graphics Forum (Proceedings of Eurographics)},
  year      = {2012},
  volume    = {31},
  number    = {2},
  month     = {May},
  year      = {2012},
  location  = {Cagliari, Italy},
  doi       = {10.1111/j.1467-8659.2012.03003.x},
  url       = {http://www.mpi-inf.mpg.de/resources/rgbz-camera/},
}