This directory contains the source code for the paper Coherent Spatiotemporal Filtering, Upsampling and Rendering of RGBZ Videos Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, Christian Theobalt Computer Graphics Forum (Proceedings of Eurographics), 31(2), May 2012 If you use our code, please cite our paper. Thanks. Copyright (c) 2012 Christian Richardt Code licence: GPL v3 ---------------------------------------------------------------------------------------------------- This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, see . Instructions for compilation ---------------------------------------------------------------------------------------------------- We use Visual Studio 2008 (SP1) on Windows 7. All our programs are compiled for 32-bit, but run on both 32-bit and 64-bit versions of Windows. All projects come with all required libraries and should compile out of the box, except for "LightingEstimationStep2" which depends on OpenCV. We used version 2.1, but any recent version should work. Available data sets ---------------------------------------------------------------------------------------------------- The data sets we show in our paper are available for download from: RGBZcamera.Recorder: capturing video streams ---------------------------------------------------------------------------------------------------- The Recorder program records video streams from our prototype camera as well as the Kinect. By default, only support for Kinect recording is compiled. Support for our prototype camera (Flea2 colour camera combined with a SwissRanger 4000 time-of-flight camera) can be enabled by including "#define CompileFleaSR4000" at the top of the "RecorderProgram.cs" file. Videos are by default recorded in the directory "data/captured/YYYY-MM-DD HH-MM-SS/" using a custom raw frame format (see below for how to read it). The video streams recorded from a Kinect are synchronised in software, so that frame numbers correspond between the depth and colour video frames. However, some frames may have been dropped by the driver or the synchronisation. Streams recorded from our prototype camera are hardware-synchronised, but often have some constant offset between frame numbers (which needs to be found out manually). Before the videos can be processed, a config file needs to be created, which describes the video streams and the corresponding camera calibration. These files are called "_config.ini" and should be located in the same directory as the video frames. A documented example of this config file can be found in the same directory like this readme file. RGBZcamera.Renderer: processing and rendering RGBZ videos ---------------------------------------------------------------------------------------------------- The Renderer application loads, processes and renders RGBZ videos in a variety of rendering styles, most of them described in our paper. Upon start, the application lists all videos it can find in the /data/input/ directory (which must be located in a parent directory of the working directory), and shows the length of each video as well as if it has been cached. The result of our RGBZ video processing pipeline can be cached, and then loaded more quickly to allow for real-time rendering. Klicking the "Start" button sets up the processing and rendering pipeline for the selected video. Using the "Settings" panel, many parameters of this pipeline can be modified at run time and the effects of these changes can be observed. Furthermore, the panel shows radiobuttons to select the rendering style, and tooggle buttons to show specific stages in our RGBZ video processing pipeline. The bottom part of the panel shows a dropdown box of all nodes in the pipeline and a list of all parameters of the selected node, which can be edited. Reading/writing raw bitmap frames ---------------------------------------------------------------------------------------------------- The Recorder and Renderer applications can use a custom raw bitmap format for video frames. This source code includes implementations to read and write such frames using C# and C++. C# (in project VideoLib): Reading from disk : VideoLib.Sources.RawSource.LoadFrameFromFile(filename) Writing to disk : VideoLib.Sinks.RawSink.SaveFrameToFile(bitmapFrame, filename, async) C++ (in project BitmapFrameReader): Reading from disk : BitmapFrames::readRawBitmapFrame(filename, header, data) Writing to disk : BitmapFrames::saveRawBitmapFrame(data, header, filename) Lighting Estimation ---------------------------------------------------------------------------------------------------- 1. First process the complete video using the "RGBZcamera.Renderer" program and make sure to tick the "Save processed video to cache" option. 2. Open the source code of the "LightingEstimationStep1" program and edit the "Main" function to use the selected dataset and to only call the "CreateNormalMapsAndShadedImages" function (and not "ExportData"). Running this program will create files named "normals-####.(png|raw)" and "shading-###.(png|raw)" in "data/output//processed/". 3. Select one video frame with clean normals and a good distribution of normal directions by browsing through the previously-generated images. 4. Next edit the "Main" function of the "LightingEstimationStep1" program again to only call the "ExportData" function, with the selected frame number. Running this program will produce a file "frame-####.txt" for the selected video frame, which contains pixel position, normal and shading information. 5. Prepare a mask image to select which image pixels contribute to the lighting estimation. White pixels contribute to the lighting estimation, but black pixels don't. The mask is useful for ignoring unreliable normals near depth discontinuities as well as self-shadowed and specular pixels (which violate the diffuse shading assumption). 6. Edit the "LightingEstimation2" program's main function (at bottom of file) to read in the right files. 7. Running the program will then produce a file "litcoef.txt" containing the estimated lighting coefficients for each of the RGB channels. 8. Select the line with the largest first coefficient and copy it into the video's _config.ini file like this: [Relighting] LightingCoefficients = 1.76, -0.14, -0.0047, 0.0093, -0.14, 0.098, 0.46, -0.044, 0.16 Contents of this directory ---------------------------------------------------------------------------------------------------- _config.ini A documented example of the configuration file used to describe all input videos. VideoLib/* Various libraries used for video processing (developed by Christian Richardt). RGBZcamera.Library The core library for our paper, which implements all RGBZ video processing and rendering styles. RGBZcamera.Recorder Our video recording tool which records multiple simultaneous video streams, e.g. from a Kinect. See the "RGBZcamera.Recorder" section above for more detail. RGBZcamera.Renderer Our video processing and rendering application which works on pre-recorded videos. See the "RGBZcamera.Renderer" section above for more detail. Tools/* Various tools for reading/writing raw bitmap frames, as well as lighting estimation. Provided for reference.