# Estimating Depth from RGB and Sparse Sensing

arXiv: https://arxiv.org/abs/1804.02771 · HTML: https://mlsanity.com/paper/1804.02771/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2018.

## Built on by (most cited first, 23 of 23)

- [Bilateral Propagation Network for Depth Completion](https://mlsanity.com/paper/2403.11270/) (2024)
- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [LRRU: Long-short Range Recurrent Updating Networks for Depth Completion](https://mlsanity.com/paper/2310.08956/) (2023)
- [Depth Completion with Twin Surface Extrapolation at Occlusion Boundaries](https://mlsanity.com/paper/2104.02253/) (2021)
- [Deep Depth Completion from Extremely Sparse Data: A Survey](https://mlsanity.com/paper/2205.05335/) (2022)
- [Metrically Scaled Monocular Depth Estimation through Sparse Priors for Underwater Robots](https://mlsanity.com/paper/2310.16750/) (2023)
- [MIPI 2022 Challenge on RGB+ToF Depth Completion: Dataset and Report](https://mlsanity.com/paper/2209.07057/) (2022)
- [A Low Memory Footprint Quantized Neural Network for Depth Completion of Very Sparse Time-of-Flight Depth Maps](https://mlsanity.com/paper/2205.12918/) (2022)
- [Differentiable Diffusion for Dense Depth Estimation from Multi-view Images](https://mlsanity.com/paper/2106.08917/) (2021)
- [MIPI 2023 Challenge on RGB+ToF Depth Completion: Methods and Results](https://mlsanity.com/paper/2304.13916/) (2023)
- [Self-Supervised Depth Completion for Active Stereo](https://mlsanity.com/paper/2110.03234/) (2021)
- [Mesh Reconstruction from Aerial Images for Outdoor Terrain Mapping Using Joint 2D-3D Learning](https://mlsanity.com/paper/2101.01844/) (2021)
- [Adaptive Illumination based Depth Sensing using Deep Superpixel and Soft Sampling Approximation](https://mlsanity.com/paper/2103.12297/) (2021)
- [How to Guide Adaptive Depth Sampling?](https://mlsanity.com/paper/2205.10202/) (2022)
- [SparseFormer: Attention-based Depth Completion Network](https://mlsanity.com/paper/2206.04557/) (2022)
- [Learning Inverse Laplacian Pyramid for Progressive Depth Completion](https://mlsanity.com/paper/2502.07289/) (2025)
- [SaccadeCam: Adaptive Visual Attention for Monocular Depth Sensing](https://mlsanity.com/paper/2103.12981/) (2021)
- [Depth-Relative Self Attention for Monocular Depth Estimation](https://mlsanity.com/paper/2304.12849/) (2023)
- [LiDAR Meta Depth Completion](https://mlsanity.com/paper/2307.12761/) (2023)
- [Improved monocular depth prediction using distance transform over pre-semantic contours with self-supervised neural networks](https://mlsanity.com/paper/2605.08320/) (2026)
- [Artificial Intelligence in Minimally Invasive Interventional Treatment](https://mlsanity.com/paper/2106.15306/) (2021)
- [High-frequency shape recovery from shading by CNN and domain adaptation](https://mlsanity.com/paper/2108.02937/) (2021)
- [TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning](https://mlsanity.com/paper/2204.10993/) (2022)

## Abstract (arXiv)

We present a deep model that can accurately produce dense depth maps given an RGB image with known depth at a very sparse set of pixels. The model works simultaneously for both indoor/outdoor scenes and produces state-of-the-art dense depth maps at nearly real-time speeds on both the NYUv2 and KITTI datasets. We surpass the state-of-the-art for monocular depth estimation even with depth values for only 1 out of every ~10000 image pixels, and we outperform other sparse-to-dense depth methods at all sparsity levels. With depth values for 1/256 of the image pixels, we achieve a mean absolute error of less than 1% of actual depth on indoor scenes, comparable to the performance of consumer-grade depth sensor hardware. Our experiments demonstrate that it would indeed be possible to efficiently transform sparse depth measurements obtained using e.g. lower-power depth sensors or SLAM systems into high-quality dense depth maps.