# Cross-spectral Gated-RGB Stereo Depth Estimation

arXiv: https://arxiv.org/abs/2405.12759 · HTML: https://mlsanity.com/paper/2405.12759/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 98 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 30 of 57)

- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation](https://mlsanity.com/paper/2203.11483/) (2022)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020)
- [Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image](https://mlsanity.com/paper/1709.07492/) (2017)
- [PENet: Towards Precise and Efficient Image Guided Depth Completion](https://mlsanity.com/paper/2103.00783/) (2021)
- [Attention Concatenation Volume for Accurate and Efficient Stereo Matching](https://mlsanity.com/paper/2203.02146/) (2022)
- [Hierarchical Deep Stereo Matching on High-resolution Images](https://mlsanity.com/paper/1912.06704/) (2019)
- [MPViT: Multi-Path Vision Transformer for Dense Prediction](https://mlsanity.com/paper/2112.11010/) (2021)
- [Attentional Feature Fusion](https://mlsanity.com/paper/2009.14082/) (2020)
- [Learning Guided Convolutional Network for Depth Completion](https://mlsanity.com/paper/1908.01238/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Self-Supervised Sparse Sensor Fusion for Long Range Perception](https://mlsanity.com/paper/2508.13995/) (2025)
- [Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision](https://mlsanity.com/paper/2411.18025/) (2024)
- [Adaptive Stereo Depth Estimation with Multi-Spectral Images Across All Lighting Conditions](https://mlsanity.com/paper/2411.03638/) (2024)
- [Iterative Volume Fusion for Asymmetric Stereo Matching](https://mlsanity.com/paper/2508.09543/) (2025)
- [SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration](https://mlsanity.com/paper/2508.12271/) (2025)
- [Ambient-robust Inverse Rendering using Active RGB-NIR Imaging](https://mlsanity.com/paper/2605.30250/) (2026)
- [Cross-Spectral Dense Correspondence for Multimodal Spectral Medical Imaging](https://mlsanity.com/paper/2608.28341/) (2026)

## Abstract (arXiv)

Gated cameras flood-illuminate a scene and capture the time-gated impulse response of a scene. By employing nanosecond-scale gates, existing sensors are capable of capturing mega-pixel gated images, delivering dense depth improving on today's LiDAR sensors in spatial resolution and depth precision. Although gated depth estimation methods deliver a million of depth estimates per frame, their resolution is still an order below existing RGB imaging methods. In this work, we combine high-resolution stereo HDR RCCB cameras with gated imaging, allowing us to exploit depth cues from active gating, multi-view RGB and multi-view NIR sensing -- multi-view and gated cues across the entire spectrum. The resulting capture system consists only of low-cost CMOS sensors and flood-illumination. We propose a novel stereo-depth estimation method that is capable of exploiting these multi-modal multi-view depth cues, including the active illumination that is measured by the RCCB camera when removing the IR-cut filter. The proposed method achieves accurate depth at long ranges, outperforming the next best existing method by 39% for ranges of 100 to 220m in MAE on accumulated LiDAR ground-truth. Our code, models and datasets are available at https://light.princeton.edu/gatedrccbstereo/ .