Deep Continuous Fusion for Multi-Sensor 3D Object Detection
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.
Key facts. Cited by 263 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 65; 2022: 77; 2023: 65; 2024: 26; 2025: 21; 2026: 9.
263 papers.
Sort:
By area:
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
- 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
- 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
- 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
- 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
- 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
- 2021OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communicationcited by 304
- 2021Voxel Transformer for 3D Object Detectioncited by 247
- 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 244
- 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
- 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
- 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
38 more
- 2021Group-Free 3D Object Detection via Transformerscited by 215
- 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 212
- 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
- 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
- 2021Multimodal Virtual Point 3D Detectioncited by 166
- 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
- 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
- 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 145
- 2021FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimationcited by 144
- 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
- 2022Learning from All Vehiclescited by 135
- 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
- 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
- 2022Vision-Centric BEV Perception: A Surveycited by 126
- 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
- 2021SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloudcited by 119
- 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 114
- 2021Improving 3D Object Detection with Channel-wise Transformercited by 108
- 20213D Object Detection for Autonomous Driving: A Surveycited by 106
- 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
- 2022Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Cloudscited by 102
- 2021Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attackscited by 101
- 2022PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detectioncited by 99
- 2021SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generationcited by 94
- 2022Surface Representation for Point Cloudscited by 84
- 2022SeedFormer: Patch Seeds based Point Cloud Completion with Upsample Transformercited by 83
- 2022Sparse Fuse Dense: Towards High Quality 3D Detection with Depth Completioncited by 82
- 2021Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimationcited by 80
- 2021Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detectioncited by 79
- 2022Multi-modal Sensor Fusion for Auto Driving Perception: A Surveycited by 76
- 2021M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformerscited by 74
- 2023FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformercited by 73
- 2022AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detectioncited by 70
- 2021Monocular 3D Object Detection: An Extrinsic Parameter Free Approachcited by 69
- 20223D Object Detection from Images for Autonomous Driving: A Surveycited by 66
- 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
- 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
- 2021Adaptive Graph Convolution for Point Cloud Analysiscited by 60
50 most cited of 263 All 263 (JSON)
- 2026From Alignment to Fusion in 3D Vision-Language23 Sep 2026
- 2026TADP: Task-Aware Deformable Prediction for Single-Stage 3D Object Detection27 Aug 2026
- 2026Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation25 Aug 2026
- 2026ProjFormer: Point Cloud Completion via Geometric-Projective Transformer and Cross-Modal Semantic Constraints15 Aug 2026
- 2026TriBand-BEV: Real-Time LiDAR-Only 3D Pedestrian Detection via Height-Aware BEV and High-Resolution Feature Fusion12 May 2026
- 2026MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning11 May 2026
- 2026UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition8 May 2026
- 2026EIMC: Efficient Instance-aware Multi-modal Collaborative Perception3 Mar 2026
- 2026Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration26 Feb 2026
- 2025ALIGN: Advanced Query Initialization with LiDAR-Image Guidance for Occlusion-Robust 3D Object Detection20 Dec 2025
- 2025DFG-PCN: Point Cloud Completion with Degree-Flexible Point Graph28 Sep 2025
- 2025SliceSemOcc: Vertical Slice Based Multimodal 3D Semantic Occupancy Representation4 Sep 2025
38 more
- 2025GMF-Drive: Gated Mamba Fusion with Spatial-Aware BEV Representation for End-to-End Autonomous Driving8 Aug 2025
- 2025EffiComm: Bandwidth Efficient Multi Agent Communication25 Jul 2025
- 2025Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion19 Jul 2025
- 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles27 Jun 2025
- 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects24 Jun 2025
- 2025Vision-based Lifting of 2D Object Detections for Automated Driving13 Jun 2025
- 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving27 May 2025
- 2025BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restoration27 May 2025
- 2025PF3Det: A Prompted Foundation Feature Assisted Visual LiDAR 3D Detector4 Apr 2025
- 2025ZFusion: An Effective Fuser of Camera and 4D Radar for 3D Object Perception in Autonomous Driving4 Apr 2025
- 2025Hyperdimensional Uncertainty Quantification for Multimodal Uncertainty Fusion in Autonomous Vehicles Perception25 Mar 2025
- 2025FrustumFusionNets: A Three-Dimensional Object Detection Network Based on Tractor Road Scene18 Mar 2025
- 2025SparseVoxFormer: Sparse Voxel-based Transformer for Multi-modal 3D Object Detection11 Mar 2025
- 2025Removing Multiple Hybrid Adverse Weather in Video via a Unified Model8 Mar 2025
- 2025Secure On-Device Video OOD Detection Without Backpropagation8 Mar 2025
- 2025DSPNet: Dual-vision Scene Perception for Robust 3D Question Answering5 Mar 2025
- 2025Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Cameras26 Feb 2025
- 2025FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection8 Jan 2025
- 2024Point Cloud Unsupervised Pre-training via 3D Gaussian Splatting27 Nov 2024
- 2024LiDAttack: Robust Black-box Attack on LiDAR-based Object Detection4 Nov 2024
- 2024Degradation-Aware Residual-Conditioned Optimal Transport for Unified Image Restoration3 Nov 2024
- 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes14 Oct 2024
- 2024MGMapNet: Multi-Granularity Representation Learning for End-to-End Vectorized HD Map Construction10 Oct 2024
- 2024Navigating Threats: A Survey of Physical Adversarial Attacks on LiDAR Perception Systems in Autonomous Vehicles30 Sep 2024
- 2024Towards Energy-Efficiency by Navigating the Trilemma of Energy, Latency, and Accuracy6 Sep 2024
- 2024TraIL-Det: Transformation-Invariant Local Feature Networks for 3D LiDAR Object Detection with Unsupervised Pre-Training25 Aug 2024
- 2024Mean Height Aided Post-Processing for Pedestrian Detection24 Aug 2024
- 2024Robust Long-Range Perception Against Sensor Misalignment in Autonomous Vehicles20 Aug 2024
- 2024GeoFormer: Learning Point Cloud Completion with Tri-Plane Integrated Transformer13 Aug 2024
- 2024Approaching Outside: Scaling Unsupervised 3D Object Detection from 2D Scene11 Jul 2024
- 2024T-CorresNet: Template Guided 3D Point Cloud Completion with Correspondence Pooling Query Generation Strategy6 Jul 2024
- 2024Is Your HD Map Constructor Reliable under Sensor Corruptions?18 Jun 2024
- 2024SpotNet: An Image Centric, Lidar Anchored Approach To Long Range Perception24 May 2024
- 2024RDPN6D: Residual-based Dense Point-wise Network for 6Dof Object Pose Estimation Based on RGB-D Images14 May 2024
- 2024RGBD-Glue: General Feature Combination for Robust RGB-D Point Cloud Registration13 May 2024
- 2024Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensors2 May 2024
- 2024A Survey on Intermediate Fusion Methods for Collaborative Perception Categorized by Real World Challenges24 Apr 2024
- 2024Weakly Supervised LiDAR Semantic Segmentation via Scatter Image Annotation19 Apr 2024
50 newest of 263 All 263 (JSON)
- 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
- 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
- 2022Learning from All Vehiclescited by 135
- 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
- 2021Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attackscited by 101
- 2022End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agentcited by 21
- 2022MMFN: Multi-Modal-Fusion-Net for End-to-End Drivingcited by 17
- 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Drivingcited by 16
- 2024M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Drivingcited by 7
- 2023Penalty-Based Imitation Learning With Cross Semantics Generation Sensor Fusion for Autonomous Drivingcited by 2
- 2021Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehiclescited by 1
- 2022Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Drivingcited by 1
12 most cited of 16 All 16 (JSON)
- 2026Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation25 Aug 2026
- 2026MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning11 May 2026
- 2025GMF-Drive: Gated Mamba Fusion with Spatial-Aware BEV Representation for End-to-End Autonomous Driving8 Aug 2025
- 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving27 May 2025
- 2024M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving19 Mar 2024
- 2023LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learning19 Oct 2023
- 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey10 Jul 2023
- 2023Penalty-Based Imitation Learning With Cross Semantics Generation Sensor Fusion for Autonomous Driving21 Mar 2023
- 2022Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving13 Oct 2022
- 2022MMFN: Multi-Modal-Fusion-Net for End-to-End Driving1 Jul 2022
- 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving31 May 2022
- 2022End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent12 Apr 2022
12 newest of 16 All 16 (JSON)
- 2023Diffusion-based 3D Object Detection with Random Boxescited by 5
- 2023Diffusion-based 3D Object Detection with Random Boxes5 Sep 2023
- 2025EffiComm: Bandwidth Efficient Multi Agent Communicationcited by 2
- 2025EffiComm: Bandwidth Efficient Multi Agent Communication25 Jul 2025
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
- 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
- 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
- 2021Voxel Transformer for 3D Object Detectioncited by 247
- 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 244
- 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
- 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
- 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
- 2021Group-Free 3D Object Detection via Transformerscited by 215
- 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 212
- 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
- 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
12 most cited of 242 All 242 (JSON)
- 2026From Alignment to Fusion in 3D Vision-Language23 Sep 2026
- 2026TADP: Task-Aware Deformable Prediction for Single-Stage 3D Object Detection27 Aug 2026
- 2026ProjFormer: Point Cloud Completion via Geometric-Projective Transformer and Cross-Modal Semantic Constraints15 Aug 2026
- 2026TriBand-BEV: Real-Time LiDAR-Only 3D Pedestrian Detection via Height-Aware BEV and High-Resolution Feature Fusion12 May 2026
- 2026UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition8 May 2026
- 2026EIMC: Efficient Instance-aware Multi-modal Collaborative Perception3 Mar 2026
- 2026Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration26 Feb 2026
- 2025ALIGN: Advanced Query Initialization with LiDAR-Image Guidance for Occlusion-Robust 3D Object Detection20 Dec 2025
- 2025DFG-PCN: Point Cloud Completion with Degree-Flexible Point Graph28 Sep 2025
- 2025SliceSemOcc: Vertical Slice Based Multimodal 3D Semantic Occupancy Representation4 Sep 2025
- 2025Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion19 Jul 2025
- 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles27 Jun 2025
12 newest of 242 All 242 (JSON)