Abhimitra Meka1    Michael Zollhöfer1    Christian Richardt1,2    Christian Theobalt1

1 MPI Informatik       2 Intel Visual Computing Institute

ACM Transactions on Graphics (Proceedings of SIGGRAPH 2016)


Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational l2–lp optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality – even without user guidance such as scribbles.



© Copyrights by the Authors, 2016. This is the authors’ version of the work. It is posted here for your personal use. Not for redistribution. The definitive version will be published in ACM Transactions on Graphics.


  author    = {Abhimitra Meka and Michael Zollh{\"o}fer and Christian Richardt and Christian Theobalt},
  title     = {Live Intrinsic Video},
  journal   = {ACM Transactions on Graphics (Proceedings of SIGGRAPH)},
  year      = {2016},
  month     = {July},
  volume    = {35},
  number    = {4},
  pages     = {109:1--14},
  doi       = {10.1145/2897824.2925907},
  url       = {http://richardt.name/live-intrinsic-video/},