# Depth on Demand: Streaming Dense Depth from a Low Frame Rate Active Sensor

arXiv: https://arxiv.org/abs/2409.08277 · HTML: https://mlsanity.com/paper/2409.08277/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 60 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [PatchmatchNet: Learned Multi-View Patchmatch Stereo](https://mlsanity.com/paper/2012.01411/) (2020)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [Learning Depth with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1810.02695/) (2018)
- [Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1808.00150/) (2018)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [CompletionFormer: Depth Completion with Convolutions and Vision Transformers](https://mlsanity.com/paper/2304.13030/) (2023)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [Dynamic Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2202.09769/) (2022)
- [Adaptive Context-Aware Multi-Modal Network for Depth Completion](https://mlsanity.com/paper/2008.10833/) (2020)
- [Bilateral Propagation Network for Depth Completion](https://mlsanity.com/paper/2403.11270/) (2024)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [Tri-Perspective View Decomposition for Geometry-Aware Depth Completion](https://mlsanity.com/paper/2403.15008/) (2024)
- [Depth Coefficients for Depth Completion](https://mlsanity.com/paper/1903.05421/) (2019)
- [VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion](https://mlsanity.com/paper/2112.00236/) (2021)
- [Propagating Confidences through CNNs for Sparse Data Regression](https://mlsanity.com/paper/1805.11913/) (2018)
- [Deep Depth Completion from Extremely Sparse Data: A Survey](https://mlsanity.com/paper/2205.05335/) (2022)

## Built on by (most cited first, 2 of 2)

- [Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation](https://mlsanity.com/paper/2412.14015/) (2024)
- [ToF-Splatting: Dense SLAM using Sparse Time-of-Flight Depth and Multi-Frame Integration](https://mlsanity.com/paper/2504.16545/) (2025)

## Abstract (arXiv)

High frame rate and accurate depth estimation plays an important role in several tasks crucial to robotics and automotive perception. To date, this can be achieved through ToF and LiDAR devices for indoor and outdoor applications, respectively. However, their applicability is limited by low frame rate, energy consumption, and spatial sparsity. Depth on Demand (DoD) allows for accurate temporal and spatial depth densification achieved by exploiting a high frame rate RGB sensor coupled with a potentially lower frame rate and sparse active depth sensor. Our proposal jointly enables lower energy consumption and denser shape reconstruction, by significantly reducing the streaming requirements on the depth sensor thanks to its three core stages: i) multi-modal encoding, ii) iterative multi-modal integration, and iii) depth decoding. We present extended evidence assessing the effectiveness of DoD on indoor and outdoor video datasets, covering both environment scanning and automotive perception use cases.