LIME: Live Intrinsic Material Estimation
Abhimitra Meka1,2 Maxim Maximov1,2 Michael Zollhöfer1,2,3 Avishek Chatterjee1,2 Hans-Peter Seidel1,2 Christian Richardt4 Christian Theobalt1,2
1 MPI Informatik 2 Saarland Informatics Campus 3 Stanford University 4 University of Bath
Computer Vision and Pattern Recognition (CVPR) 2018 — Spotlight presentation
Abstract
We present the first end-to-end approach for real-time material estimation for general object shapes with uniform material that only requires a single color image as input. In addition to Lambertian surface properties, our approach fully automatically computes the specular albedo, material shininess, and a foreground segmentation. We tackle this challenging and ill-posed inverse rendering problem using recent advances in image-to-image translation techniques based on deep convolutional encoder–decoder architectures. The underlying core representations of our approach are specular shading, diffuse shading and mirror images, which allow to learn the effective and accurate separation of diffuse and specular albedo. In addition, we propose a novel highly efficient perceptual rendering loss that mimics real-world image formation and obtains intermediate results even during run time. The estimation of material parameters at real-time frame rates enables exciting mixed-reality applications, such as seamless illumination-consistent integration of virtual objects into real-world scenes, and virtual material cloning. We demonstrate our approach in a live setup, compare it to the state of the art, and demonstrate its effectiveness through quantitative and qualitative evaluation.
Downloads
- Paper preprint (PDF, 7 MB)
- Supplemental document (PDF, 3 MB)
- Supplemental video (MP4, 81 MB)
- This paper on arXiv (arXiv:1801.01075)
- Additional materials on GVV project website
Copyright
© Copyrights by the Authors, 2018. This is the authors’ version of the work. It is posted here for your personal use. Not for redistribution. The definitive version will be published in IEEE Xplore.
Bibtex
@inproceedings{LIME, author = {Abhimitra Meka and Maxim Maximov and Michael Zollhöfer and Avishek Chatterjee and Christian Richardt and Christian Theobalt}, title = {{LIME}: Live Intrinsic Material Estimation}, booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2018}, month = jun, pages = {6315--6324}, doi = {10.1109/CVPR.2018.00661}, url = {http://richardt.name/publications/LIME/}, }