# Radar-Camera Pixel Depth Association for Depth Completion

arXiv: https://arxiv.org/abs/2106.02778 · HTML: https://mlsanity.com/paper/2106.02778/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 5 Jun 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image](https://mlsanity.com/paper/1709.07492/) (2017)
- [CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2011.04841/) (2020)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset](https://mlsanity.com/paper/1909.01300/) (2019)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [Self-Supervised Monocular Depth Hints](https://mlsanity.com/paper/1909.09051/) (2019)
- [Kinematic 3D Object Detection in Monocular Video](https://mlsanity.com/paper/2007.09548/) (2020)
- [Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints](https://mlsanity.com/paper/1910.06727/) (2019)
- [SfM-Net: Learning of Structure and Motion from Video](https://mlsanity.com/paper/1704.07804/) (2017)
- [Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation](https://mlsanity.com/paper/1808.00769/) (2018)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection](https://mlsanity.com/paper/2005.07431/) (2020)
- [Distant Vehicle Detection Using Radar and Vision](https://mlsanity.com/paper/1901.10951/) (2019)
- [Depth Coefficients for Depth Completion](https://mlsanity.com/paper/1903.05421/) (2019)
- [Depth Completion with Twin Surface Extrapolation at Occlusion Boundaries](https://mlsanity.com/paper/2104.02253/) (2021)

## Built on by (most cited first, 30 of 50)

- [CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception](https://mlsanity.com/paper/2304.00670/) (2023)
- [CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformer](https://mlsanity.com/paper/2209.06535/) (2022)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes](https://mlsanity.com/paper/2108.04814/) (2021)
- [Directional Connectivity-based Segmentation of Medical Images](https://mlsanity.com/paper/2304.00145/) (2023)
- [Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2312.04861/) (2023)
- [ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions](https://mlsanity.com/paper/2210.01346/) (2022)
- [RadarCam-Depth: Radar-Camera Fusion for Depth Estimation with Learned Metric Scale](https://mlsanity.com/paper/2401.04325/) (2024)
- [DART: Implicit Doppler Tomography for Radar Novel View Synthesis](https://mlsanity.com/paper/2403.03896/) (2024)
- [HVDetFusion: A Simple and Robust Camera-Radar Fusion Framework](https://mlsanity.com/paper/2307.11323/) (2023)
- [Full-Velocity Radar Returns by Radar-Camera Fusion](https://mlsanity.com/paper/2108.10637/) (2021)
- [Sparse Beats Dense: Rethinking Supervision in Radar-Camera Depth Completion](https://mlsanity.com/paper/2312.00844/) (2023)
- [CaFNet: A Confidence-Driven Framework for Radar Camera Depth Estimation](https://mlsanity.com/paper/2407.00697/) (2024)
- [MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources](https://mlsanity.com/paper/2601.22054/) (2026)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [RICCARDO: Radar Hit Prediction and Convolution for Camera-Radar 3D Object Detection](https://mlsanity.com/paper/2504.09086/) (2025)
- [TacoDepth: Towards Efficient Radar-Camera Depth Estimation with One-stage Fusion](https://mlsanity.com/paper/2504.11773/) (2025)
- [RCDPT: Radar-Camera fusion Dense Prediction Transformer](https://mlsanity.com/paper/2211.02432/) (2022)
- [Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile Vision](https://mlsanity.com/paper/2109.08880/) (2021)
- [GET-UP: GEomeTric-aware Depth Estimation with Radar Points UPsampling](https://mlsanity.com/paper/2409.02720/) (2024)
- [ZFusion: An Effective Fuser of Camera and 4D Radar for 3D Object Perception in Autonomous Driving](https://mlsanity.com/paper/2504.03438/) (2025)
- [Advancing Autonomous Driving: DepthSense with Radar and Spatial Attention](https://mlsanity.com/paper/2109.05265/) (2021)
- [RIDERS: Radar-Infrared Depth Estimation for Robust Sensing](https://mlsanity.com/paper/2402.02067/) (2024)
- [LiRCDepth: Lightweight Radar-Camera Depth Estimation via Knowledge Distillation and Uncertainty Guidance](https://mlsanity.com/paper/2412.16380/) (2024)
- [Radar-Guided Polynomial Fitting for Metric Depth Estimation](https://mlsanity.com/paper/2503.17182/) (2025)
- [Structure-Aware Radar-Camera Depth Estimation](https://mlsanity.com/paper/2506.05008/) (2025)
- [Deep Depth Estimation from Thermal Image: Dataset, Benchmark, and Challenges](https://mlsanity.com/paper/2503.22060/) (2025)
- [RC-AutoCalib: An End-to-End Radar-Camera Automatic Calibration Network](https://mlsanity.com/paper/2505.22427/) (2025)
- [Bridging Spectral-wise and Multi-spectral Depth Estimation via Geometry-guided Contrastive Learning](https://mlsanity.com/paper/2503.00793/) (2025)
- [How Much Depth Information can Radar Contribute to a Depth Estimation Model?](https://mlsanity.com/paper/2202.13220/) (2022)

## Abstract (arXiv)

While radar and video data can be readily fused at the detection level, fusing them at the pixel level is potentially more beneficial. This is also more challenging in part due to the sparsity of radar, but also because automotive radar beams are much wider than a typical pixel combined with a large baseline between camera and radar, which results in poor association between radar pixels and color pixel. A consequence is that depth completion methods designed for LiDAR and video fare poorly for radar and video. Here we propose a radar-to-pixel association stage which learns a mapping from radar returns to pixels. This mapping also serves to densify radar returns. Using this as a first stage, followed by a more traditional depth completion method, we are able to achieve image-guided depth completion with radar and video. We demonstrate performance superior to camera and radar alone on the nuScenes dataset. Our source code is available at https://github.com/longyunf/rc-pda.