# BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation

arXiv: https://arxiv.org/abs/2403.11761 · HTML: https://mlsanity.com/paper/2403.11761/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset](https://mlsanity.com/paper/1909.01300/) (2019)
- [Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?](https://mlsanity.com/paper/2206.07959/) (2022)
- [BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs](https://mlsanity.com/paper/2203.04050/) (2022)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks](https://mlsanity.com/paper/1804.02176/) (2018)
- [CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception](https://mlsanity.com/paper/2304.00670/) (2023)
- [TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving](https://mlsanity.com/paper/2204.13483/) (2022)
- [FISHING Net: Future Inference of Semantic Heatmaps In Grids](https://mlsanity.com/paper/2006.09917/) (2020)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [Few-Shot Panoptic Segmentation With Foundation Models](https://mlsanity.com/paper/2309.10726/) (2023)
- [Semantic Scene Segmentation for Robotics](https://mlsanity.com/paper/2401.07589/) (2024)

## Built on by (most cited first, 27 of 27)

- [RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network](https://mlsanity.com/paper/2409.04979/) (2024)
- [Bridging Perspectives: Foundation Model Guided BEV Maps for 3D Object Detection and Tracking](https://mlsanity.com/paper/2510.10287/) (2025)
- [Multi-Scale Neighborhood Occupancy Masked Autoencoder for Self-Supervised Learning in LiDAR Point Clouds](https://mlsanity.com/paper/2502.20316/) (2025)
- [Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2509.20107/) (2025)
- [Robust Bird's Eye View Segmentation by Adapting DINOv2](https://mlsanity.com/paper/2409.10228/) (2024)
- [UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation](https://mlsanity.com/paper/2602.19349/) (2026)
- [FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras](https://mlsanity.com/paper/2509.13681/) (2025)
- [FishRoPE: Projective Rotary Position Embeddings for Omnidirectional Visual Perception](https://mlsanity.com/paper/2604.10391/) (2026)
- [CaRaFFusion: Improving 2D Semantic Segmentation with Camera-Radar Point Cloud Fusion and Zero-Shot Image Inpainting](https://mlsanity.com/paper/2505.03679/) (2025)
- [RESAR-BEV: An Explainable Progressive Residual Autoregressive Approach for Camera-Radar Fusion in BEV Segmentation](https://mlsanity.com/paper/2505.06515/) (2025)
- [MapFM: Foundation Model-Driven HD Mapping with Multi-Task Contextual Learning](https://mlsanity.com/paper/2506.15313/) (2025)
- [LetsMap: Unsupervised Representation Learning for Semantic BEV Mapping](https://mlsanity.com/paper/2405.18852/) (2024)
- [Velocity Driven Vision: Asynchronous Sensor Fusion Birds Eye View Models for Autonomous Vehicles](https://mlsanity.com/paper/2407.16636/) (2024)
- [PTQAT: A Hybrid Parameter-Efficient Quantization Algorithm for 3D Perception Tasks](https://mlsanity.com/paper/2508.10557/) (2025)
- [Occluded nuScenes: A Multi-Sensor Dataset for Evaluating Perception Robustness in Automated Driving](https://mlsanity.com/paper/2510.18552/) (2025)
- [MambaFusion: Adaptive State-Space Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2602.08126/) (2026)
- [RESBev: Making BEV Perception More Robust](https://mlsanity.com/paper/2603.09529/) (2026)
- [RCDINO: Enhancing Radar-Camera 3D Object Detection with DINOv2 Semantic Features](https://mlsanity.com/paper/2508.15353/) (2025)
- [FIN: Fast Inference Network for Map Segmentation](https://mlsanity.com/paper/2510.00651/) (2025)
- [IMKD: Intensity-Aware Multi-Level Knowledge Distillation for Camera-Radar Fusion](https://mlsanity.com/paper/2512.15581/) (2025)
- [DinoRADE: Full Spectral Radar-Camera Fusion with Vision Foundation Model Features for Multi-class Object Detection in Adverse Weather](https://mlsanity.com/paper/2604.08074/) (2026)
- [Radar-Camera BEV Multi-Task Learning with Cross-Task Attention Bridge for Joint 3D Detection and Segmentation](https://mlsanity.com/paper/2604.12918/) (2026)
- [CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion](https://mlsanity.com/paper/2604.24044/) (2026)
- [Joint Target-Less Intrinsic and Extrinsic Camera-LiDAR Calibration using Deep Point Correspondences](https://mlsanity.com/paper/2605.23397/) (2026)
- [Neuromorphic LiDAR-based Bird's Eye View Object Detection using Energy-efficient Spiking Neural Networks](https://mlsanity.com/paper/2605.25293/) (2026)
- [CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining](https://mlsanity.com/paper/2607.04541/) (2026)
- [Multi-Modal Traffic Sign Detection with Semantic Attributes for Autonomous Driving](https://mlsanity.com/paper/2608.20874/) (2026)

## Abstract (arXiv)

Semantic scene segmentation from a bird's-eye-view (BEV) perspective plays a crucial role in facilitating planning and decision-making for mobile robots. Although recent vision-only methods have demonstrated notable advancements in performance, they often struggle under adverse illumination conditions such as rain or nighttime. While active sensors offer a solution to this challenge, the prohibitively high cost of LiDARs remains a limiting factor. Fusing camera data with automotive radars poses a more inexpensive alternative but has received less attention in prior research. In this work, we aim to advance this promising avenue by introducing BEVCar, a novel approach for joint BEV object and map segmentation. The core novelty of our approach lies in first learning a point-based encoding of raw radar data, which is then leveraged to efficiently initialize the lifting of image features into the BEV space. We perform extensive experiments on the nuScenes dataset and demonstrate that BEVCar outperforms the current state of the art. Moreover, we show that incorporating radar information significantly enhances robustness in challenging environmental conditions and improves segmentation performance for distant objects. To foster future research, we provide the weather split of the nuScenes dataset used in our experiments, along with our code and trained models at http://bevcar.cs.uni-freiburg.de.