# Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera

arXiv: https://arxiv.org/abs/1901.02571 · HTML: https://mlsanity.com/paper/1901.02571/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 30 of 69)

- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo](https://mlsanity.com/paper/2109.01129/) (2021)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](https://mlsanity.com/paper/2204.00987/) (2022)
- [Geometry Uncertainty Projection Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13774/) (2021)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [Neural 3D Reconstruction in the Wild](https://mlsanity.com/paper/2205.12955/) (2022)
- [FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent](https://mlsanity.com/paper/2404.15259/) (2024)
- [Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry](https://mlsanity.com/paper/2112.08177/) (2021)
- [A Benchmark and a Baseline for Robust Multi-view Depth Estimation](https://mlsanity.com/paper/2209.06681/) (2022)
- [Monocular 3D Object Detection with Depth from Motion](https://mlsanity.com/paper/2207.12988/) (2022)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2110.08192/) (2021)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [Temporally Consistent Online Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2111.09337/) (2021)
- [MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation](https://mlsanity.com/paper/2307.14336/) (2023)
- [Binary TTC: A Temporal Geofence for Autonomous Navigation](https://mlsanity.com/paper/2101.04777/) (2021)
- [Is my Depth Ground-Truth Good Enough? HAMMER -- Highly Accurate Multi-Modal Dataset for DEnse 3D Scene Regression](https://mlsanity.com/paper/2205.04565/) (2022)
- [MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision](https://mlsanity.com/paper/2303.03315/) (2023)
- [Monocular Visual-Inertial Depth Estimation](https://mlsanity.com/paper/2303.12134/) (2023)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [Depth Field Networks for Generalizable Multi-view Scene Representation](https://mlsanity.com/paper/2207.14287/) (2022)
- [MonoNeuralFusion: Online Monocular Neural 3D Reconstruction with Geometric Priors](https://mlsanity.com/paper/2209.15153/) (2022)
- [Real-Time Dense 3D Mapping of Underwater Environments](https://mlsanity.com/paper/2304.02704/) (2023)
- [DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium](https://mlsanity.com/paper/2304.03560/) (2023)
- [FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution](https://mlsanity.com/paper/2504.07093/) (2025)
- [One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception](https://mlsanity.com/paper/2306.12681/) (2023)
- [FutureDepth: Learning to Predict the Future Improves Video Depth Estimation](https://mlsanity.com/paper/2403.12953/) (2024)

## Abstract (arXiv)

Depth sensing is crucial for 3D reconstruction and scene understanding. Active depth sensors provide dense metric measurements, but often suffer from limitations such as restricted operating ranges, low spatial resolution, sensor interference, and high power consumption. In this paper, we propose a deep learning (DL) method to estimate per-pixel depth and its uncertainty continuously from a monocular video stream, with the goal of effectively turning an RGB camera into an RGB-D camera. Unlike prior DL-based methods, we estimate a depth probability distribution for each pixel rather than a single depth value, leading to an estimate of a 3D depth probability volume for each input frame. These depth probability volumes are accumulated over time under a Bayesian filtering framework as more incoming frames are processed sequentially, which effectively reduces depth uncertainty and improves accuracy, robustness, and temporal stability. Compared to prior work, the proposed approach achieves more accurate and stable results, and generalizes better to new datasets. Experimental results also show the output of our approach can be directly fed into classical RGB-D based 3D scanning methods for 3D scene reconstruction.