# RIDERS: Radar-Infrared Depth Estimation for Robust Sensing

arXiv: https://arxiv.org/abs/2402.02067 · HTML: https://mlsanity.com/paper/2402.02067/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [Learning to Recover 3D Scene Shape from a Single Image](https://mlsanity.com/paper/2012.09365/) (2020)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving](https://mlsanity.com/paper/2204.13483/) (2022)
- [RadarScenes: A Real-World Radar Point Cloud Data Set for Automotive Applications](https://mlsanity.com/paper/2104.02493/) (2021)
- [Robust Monocular Depth Estimation under Challenging Conditions](https://mlsanity.com/paper/2308.09711/) (2023)
- [Radar-Camera Pixel Depth Association for Depth Completion](https://mlsanity.com/paper/2106.02778/) (2021)
- [Depth Estimation from Monocular Images and Sparse Radar Data](https://mlsanity.com/paper/2010.00058/) (2020)
- [Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation](https://mlsanity.com/paper/2108.07628/) (2021)
- [Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image](https://mlsanity.com/paper/2208.13241/) (2022)
- [R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes](https://mlsanity.com/paper/2108.04814/) (2021)
- [Depth Estimation from Monocular Images and Sparse radar using Deep Ordinal Regression Network](https://mlsanity.com/paper/2107.07596/) (2021)
- [RadarCam-Depth: Radar-Camera Fusion for Depth Estimation with Learned Metric Scale](https://mlsanity.com/paper/2401.04325/) (2024)
- [Monocular Visual-Inertial Depth Estimation](https://mlsanity.com/paper/2303.12134/) (2023)
- [NTU4DRadLM: 4D Radar-centric Multi-Modal Dataset for Localization and Mapping](https://mlsanity.com/paper/2309.00962/) (2023)
- [Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Loss](https://mlsanity.com/paper/2103.00760/) (2021)
- [Full-Velocity Radar Returns by Radar-Camera Fusion](https://mlsanity.com/paper/2108.10637/) (2021)
- [Maximizing Self-supervision from Thermal Image for Effective Self-supervised Learning of Depth and Ego-motion](https://mlsanity.com/paper/2201.04387/) (2022)
- [NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields](https://mlsanity.com/paper/2312.13471/) (2023)

## Built on by (most cited first, 5 of 5)

- [Caltech Aerial RGB-Thermal Dataset in the Wild](https://mlsanity.com/paper/2403.08997/) (2024)
- [LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera](https://mlsanity.com/paper/2407.16197/) (2024)
- [Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios](https://mlsanity.com/paper/2402.11826/) (2024)
- [4D mmWave Radar for Sensing Enhancement in Adverse Environments: Advances and Challenges](https://mlsanity.com/paper/2503.24091/) (2025)
- [Hydra: Accurate Multi-Modal Leaf Wetness Sensing with mm-Wave and Camera Fusion](https://mlsanity.com/paper/2508.02409/) (2025)

## Abstract (arXiv)

Dense depth recovery is crucial in autonomous driving, serving as a foundational element for obstacle avoidance, 3D object detection, and local path planning. Adverse weather conditions, including haze, dust, rain, snow, and darkness, introduce significant challenges to accurate dense depth estimation, thereby posing substantial safety risks in autonomous driving. These challenges are particularly pronounced for traditional depth estimation methods that rely on short electromagnetic wave sensors, such as visible spectrum cameras and near-infrared LiDAR, due to their susceptibility to diffraction noise and occlusion in such environments. To fundamentally overcome this issue, we present a novel approach for robust metric depth estimation by fusing a millimeter-wave Radar and a monocular infrared thermal camera, which are capable of penetrating atmospheric particles and unaffected by lighting conditions. Our proposed Radar-Infrared fusion method achieves highly accurate and finely detailed dense depth estimation through three stages, including monocular depth prediction with global scale alignment, quasi-dense Radar augmentation by learning Radar-pixels correspondences, and local scale refinement of dense depth using a scale map learner. Our method achieves exceptional visual quality and accurate metric estimation by addressing the challenges of ambiguity and misalignment that arise from directly fusing multi-modal long-wave features. We evaluate the performance of our approach on the NTU4DRadLM dataset and our self-collected challenging ZJU-Multispectrum dataset. Especially noteworthy is the unprecedented robustness demonstrated by our proposed method in smoky scenarios. Our code will be released at urlhttps://github.com/MMOCKING/RIDERS.