# RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization

arXiv: https://arxiv.org/abs/2102.05150 · HTML: https://mlsanity.com/paper/2102.05150/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 67 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 30 of 38)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset](https://mlsanity.com/paper/1909.01300/) (2019)
- [CNN based Road User Detection using the 3D Radar Cube](https://mlsanity.com/paper/2004.12165/) (2020)
- [Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar](https://mlsanity.com/paper/1912.06613/) (2019)
- [Exploit the Connectivity: Multi-Object Tracking with TrackletNet](https://mlsanity.com/paper/1811.07258/) (2018)

## Built on by (most cited first, 30 of 49)

- [Raw High-Definition Radar for Multi-Task Learning](https://mlsanity.com/paper/2112.10646/) (2021)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [SMURF: Spatial Multi-Representation Fusion for 3D Object Detection with 4D Imaging Radar](https://mlsanity.com/paper/2307.10784/) (2023)
- [WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces](https://mlsanity.com/paper/2307.06505/) (2023)
- [Multi-View Radar Semantic Segmentation](https://mlsanity.com/paper/2103.16214/) (2021)
- [RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar](https://mlsanity.com/paper/2405.14014/) (2024)
- [Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion](https://mlsanity.com/paper/2307.16532/) (2023)
- [Rethinking of Radar's Role: A Camera-Radar Dataset and Systematic Annotator via Coordinate Alignment](https://mlsanity.com/paper/2105.05207/) (2021)
- [Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2312.04861/) (2023)
- [MVFAN: Multi-View Feature Assisted Network for 4D Radar Object Detection](https://mlsanity.com/paper/2310.16389/) (2023)
- [DPFT: Dual Perspective Fusion Transformer for Camera-Radar-based Object Detection](https://mlsanity.com/paper/2404.03015/) (2024)
- [Towards Dense and Accurate Radar Perception Via Efficient Cross-Modal Diffusion Model](https://mlsanity.com/paper/2403.08460/) (2024)
- [RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark](https://mlsanity.com/paper/2407.13930/) (2024)
- [RadSimReal: Bridging the Gap Between Synthetic and Real Data in Radar Object Detection With Simulation](https://mlsanity.com/paper/2404.18150/) (2024)
- [LiRaFusion: Deep Adaptive LiDAR-Radar Fusion for 3D Object Detection](https://mlsanity.com/paper/2402.11735/) (2024)
- [MMVR: Millimeter-wave Multi-View Radar Dataset and Benchmark for Indoor Perception](https://mlsanity.com/paper/2406.10708/) (2024)
- [A recurrent CNN for online object detection on raw radar frames](https://mlsanity.com/paper/2212.11172/) (2022)
- [CenterRadarNet: Joint 3D Object Detection and Tracking Framework using 4D FMCW Radar](https://mlsanity.com/paper/2311.01423/) (2023)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [Self-Supervised Radio-Visual Representation Learning for 6G Sensing](https://mlsanity.com/paper/2111.02887/) (2021)
- [Improved Multi-Scale Grid Rendering of Point Clouds for Radar Object Detection Networks](https://mlsanity.com/paper/2305.15836/) (2023)
- [Conformal Distributed Remote Inference in Sensor Networks Under Reliability and Communication Constraints](https://mlsanity.com/paper/2409.07902/) (2024)
- [ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion](https://mlsanity.com/paper/2405.05164/) (2024)
- [SparseRadNet: Sparse Perception Neural Network on Subsampled Radar Data](https://mlsanity.com/paper/2406.10600/) (2024)
- [Improving Uncertainty of Deep Learning-based Object Classification on Radar Spectra using Label Smoothing](https://mlsanity.com/paper/2109.12851/) (2021)
- [Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous Driving](https://mlsanity.com/paper/2311.10261/) (2023)
- [4D Radar Semantic Segmentation of People in Field Conditions Using Temporal Multi-View Networks](https://mlsanity.com/paper/2404.05307/) (2024)
- [Wavelet-based Multi-View Fusion of 4D Radar Tensor and Camera for Robust 3D Object Detection](https://mlsanity.com/paper/2512.22972/) (2025)
- [Look, Radiate, and Learn: Self-Supervised Localisation via Radio-Visual Correspondence](https://mlsanity.com/paper/2206.06424/) (2022)
- [End-to-end system for object detection from sub-sampled radar data](https://mlsanity.com/paper/2203.03905/) (2022)

## Abstract (arXiv)

Various autonomous or assisted driving strategies have been facilitated through the accurate and reliable perception of the environment around a vehicle. Among the commonly used sensors, radar has usually been considered as a robust and cost-effective solution even in adverse driving scenarios, e.g., weak/strong lighting or bad weather. Instead of considering to fuse the unreliable information from all available sensors, perception from pure radar data becomes a valuable alternative that is worth exploring. In this paper, we propose a deep radar object detection network, named RODNet, which is cross-supervised by a camera-radar fused algorithm without laborious annotation efforts, to effectively detect objects from the radio frequency (RF) images in real-time. First, the raw signals captured by millimeter-wave radars are transformed to RF images in range-azimuth coordinates. Second, our proposed RODNet takes a sequence of RF images as the input to predict the likelihood of objects in the radar field of view (FoV). Two customized modules are also added to handle multi-chirp information and object relative motion. Instead of using human-labeled ground truth for training, the proposed RODNet is cross-supervised by a novel 3D localization of detected objects using a camera-radar fusion (CRF) strategy in the training stage. Finally, we propose a method to evaluate the object detection performance of the RODNet. Due to no existing public dataset available for our task, we create a new dataset, named CRUW, which contains synchronized RGB and RF image sequences in various driving scenarios. With intensive experiments, our proposed cross-supervised RODNet achieves 86% average precision and 88% average recall of object detection performance, which shows the robustness to noisy scenarios in various driving conditions.