# RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion

arXiv: https://arxiv.org/abs/2412.12725 · HTML: https://mlsanity.com/paper/2412.12725/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [Unifying Voxel-based Representation with Transformer for 3D Object Detection](https://mlsanity.com/paper/2206.00630/) (2022)
- [An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection](https://mlsanity.com/paper/1904.09730/) (2019)
- [Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion](https://mlsanity.com/paper/2211.10581/) (2022)
- [CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2011.04841/) (2020)
- [SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videos](https://mlsanity.com/paper/2308.09244/) (2023)
- [PolarFormer: Multi-camera 3D Object Detection with Polar Transformer](https://mlsanity.com/paper/2206.15398/) (2022)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [3D Object Detection for Autonomous Driving: A Survey](https://mlsanity.com/paper/2106.10823/) (2021)
- [FB-BEV: BEV Representation from Forward-Backward View Transformations](https://mlsanity.com/paper/2308.02236/) (2023)
- [CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception](https://mlsanity.com/paper/2304.00670/) (2023)

## Built on by (most cited first, 19 of 19)

- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception](https://mlsanity.com/paper/2501.15394/) (2025)
- [RaGS: Unleashing 3D Gaussian Splatting from 4D Radar and Monocular Cues for 3D Object Detection](https://mlsanity.com/paper/2507.19856/) (2025)
- [RQR3D: Reparametrizing the regression targets for BEV-based 3D object detection](https://mlsanity.com/paper/2505.17732/) (2025)
- [Controllable Radar Simulation with Waveform Parameter Embedding](https://mlsanity.com/paper/2506.03134/) (2025)
- [RLM: A Vision-Language Model Approach for Radar Scene Understanding](https://mlsanity.com/paper/2511.21105/) (2025)
- [RC-GeoCP: Geometric Consensus for 4D Radar-Camera Collaborative Perception](https://mlsanity.com/paper/2603.00654/) (2026)
- [SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving](https://mlsanity.com/paper/2508.10567/) (2025)
- [M^3Detection: Multi-Frame Multi-Level Feature Fusion for Multi-Modal 3D Object Detection with Camera and 4D Imaging Radar](https://mlsanity.com/paper/2510.27166/) (2025)
- [Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation](https://mlsanity.com/paper/2605.11840/) (2026)
- [RCDINO: Enhancing Radar-Camera 3D Object Detection with DINOv2 Semantic Features](https://mlsanity.com/paper/2508.15353/) (2025)
- [Radar-Camera BEV Multi-Task Learning with Cross-Task Attention Bridge for Joint 3D Detection and Segmentation](https://mlsanity.com/paper/2604.12918/) (2026)
- [Control Your Queries: Heterogeneous Query Interaction for Camera-Radar Fusion](https://mlsanity.com/paper/2604.25574/) (2026)
- [GraphBEV++: Multi-Modal Feature Alignment for Autonomous Driving](https://mlsanity.com/paper/2606.16354/) (2026)
- [Horizon3D: Sparse Radar-Camera Fusion for Long-Range 3D Perception in Autonomous Driving](https://mlsanity.com/paper/2606.31096/) (2026)
- [Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion](https://mlsanity.com/paper/2607.04098/) (2026)
- [RAF: Reliability-Aware Fusion of Camera, LiDAR, and 4D RADAR for Robust 3D Object Detection in Adverse Weather](https://mlsanity.com/paper/2607.04587/) (2026)
- [RLG-TPV: Radar- and LiDAR-Guided Tri-Perspective View Fusion for Camera-Radar 3D Object Detection](https://mlsanity.com/paper/2608.29194/) (2026)
- [SGDet3D++: Geometry-Grounded Semantics for 4D Radar and Camera 3D Object Detection](https://mlsanity.com/paper/2609.27671/) (2026)

## Abstract (arXiv)

We propose Radar-Camera fusion transformer (RaCFormer) to boost the accuracy of 3D object detection by the following insight. The Radar-Camera fusion in outdoor 3D scene perception is capped by the image-to-BEV transformation--if the depth of pixels is not accurately estimated, the naive combination of BEV features actually integrates unaligned visual content. To avoid this problem, we propose a query-based framework that enables adaptive sampling of instance-relevant features from both the bird's-eye view (BEV) and the original image view. Furthermore, we enhance system performance by two key designs: optimizing query initialization and strengthening the representational capacity of BEV. For the former, we introduce an adaptive circular distribution in polar coordinates to refine the initialization of object queries, allowing for a distance-based adjustment of query density. For the latter, we initially incorporate a radar-guided depth head to refine the transformation from image view to BEV. Subsequently, we focus on leveraging the Doppler effect of radar and introduce an implicit dynamic catcher to capture the temporal elements within the BEV. Extensive experiments on nuScenes and View-of-Delft (VoD) datasets validate the merits of our design. Remarkably, our method achieves superior results of 64.9% mAP and 70.2% NDS on nuScenes. RaCFormer also secures the state-of-the-art performance on the VoD dataset. Code is available at https://github.com/cxmomo/RaCFormer.