# High-Resolution Depth Maps Based on TOF-Stereo Fusion

arXiv: https://arxiv.org/abs/2107.14688 · HTML: https://mlsanity.com/paper/2107.14688/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 23 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Built on by (most cited first, 5 of 5)

- [Active Stereo Without Pattern Projector](https://mlsanity.com/paper/2309.12315/) (2023)
- [LiDAR-Event Stereo Fusion with Hallucinations](https://mlsanity.com/paper/2408.04633/) (2024)
- [LiDAR-guided Stereo Matching with a Spatial Consistency Constraint](https://mlsanity.com/paper/2202.09953/) (2022)
- [Joint Depth and Normal Estimation from Real-world Time-of-flight Raw Data](https://mlsanity.com/paper/2108.03649/) (2021)
- [Active Stereo in the Wild through Virtual Pattern Projection](https://mlsanity.com/paper/2406.04345/) (2024)

## Abstract (arXiv)

The combination of range sensors with color cameras can be very useful for robot navigation, semantic perception, manipulation, and telepresence. Several methods of combining range- and color-data have been investigated and successfully used in various robotic applications. Most of these systems suffer from the problems of noise in the range-data and resolution mismatch between the range sensor and the color cameras, since the resolution of current range sensors is much less than the resolution of color cameras. High-resolution depth maps can be obtained using stereo matching, but this often fails to construct accurate depth maps of weakly/repetitively textured scenes, or if the scene exhibits complex self-occlusions. Range sensors provide coarse depth information regardless of presence/absence of texture. The use of a calibrated system, composed of a time-of-flight (TOF) camera and of a stereoscopic camera pair, allows data fusion thus overcoming the weaknesses of both individual sensors. We propose a novel TOF-stereo fusion method based on an efficient seed-growing algorithm which uses the TOF data projected onto the stereo image pair as an initial set of correspondences. These initial "seeds" are then propagated based on a Bayesian model which combines an image similarity score with rough depth priors computed from the low-resolution range data. The overall result is a dense and accurate depth map at the resolution of the color cameras at hand. We show that the proposed algorithm outperforms 2D image-based stereo algorithms and that the results are of higher resolution than off-the-shelf color-range sensors, e.g., Kinect. Moreover, the algorithm potentially exhibits real-time performance on a single CPU.