mlsanityData to 7 Oct 2026

FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

arXiv 2203.10642 ↗First posted 20 Mar 2022cs.CVcited by 194

Not on the map. Why?

Abstract (arXiv)

Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (58.0 mAP) achieves on par performance with state-of-the-art 3D detection model CenterPoint (56.6 mAP) using a 32-beam LiDAR.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 20 Mar 2022.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  9. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  10. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  11. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  12. 2019Objects as Pointscited by 1,172
29 more
  1. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  2. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  3. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  4. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  5. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  6. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  7. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  8. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  9. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  10. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  11. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  12. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  13. 2021Is Pseudo-Lidar needed for Monocular 3D Object detection?cited by 248
  14. 2019Disentangling Monocular 3D Object Detectioncited by 215
  15. 2021Probabilistic and Geometric Depth: Detecting Objects in Perspectivecited by 190
  16. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  17. 2021Multimodal Virtual Point 3D Detectioncited by 166
  18. 2020CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detectioncited by 159
  19. 2021RangeDet:In Defense of Range View for LiDAR-based 3D Object Detectioncited by 140
  20. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  21. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  22. 2019Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detectioncited by 124
  23. 2021RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detectioncited by 115
  24. 2016Vehicle Detection from 3D Lidar Using Fully Convolutional Networkcited by 105
  25. 2021FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detectioncited by 92
  26. 2020A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detectioncited by 70
  27. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  28. 2019Distant Vehicle Detection Using Radar and Visioncited by 44
  29. 2021To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernelscited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 53; 2024: 55; 2025: 45; 2026: 23.

194 papers.

Sort:

By area:

  1. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  4. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  5. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  6. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  7. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  8. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  9. 2022Vision-Centric BEV Perception: A Surveycited by 126
  10. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  11. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  12. 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
38 more
  1. 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
  2. 2023Virtual Sparse Convolution for Multimodal 3D Object Detectioncited by 81
  3. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  4. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  5. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  6. 2023FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformercited by 72
  7. 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 68
  8. 2023Neural Map Prior for Autonomous Drivingcited by 68
  9. 2022CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformercited by 66
  10. 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
  11. 2023FocalFormer3D : Focusing on Hard Instance for 3D Object Detectioncited by 58
  12. 2023Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Mapscited by 58
  13. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 57
  14. 2023LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusioncited by 57
  15. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  16. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  17. 2024DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Inputcited by 53
  18. 2022Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detectioncited by 49
  19. 2022SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generationcited by 39
  20. 2024Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Drivingcited by 39
  21. 2022X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentationcited by 38
  22. 2023EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detectioncited by 38
  23. 2023Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Drivingcited by 37
  24. 2024Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perceptioncited by 35
  25. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  26. 2024GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detectioncited by 33
  27. 2023UniWorld: Autonomous Driving Pre-training via World Modelscited by 32
  28. 2024OccFusion: Multi-Sensor Fusion Framework for 3D Semantic Occupancy Predictioncited by 32
  29. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  30. 2023BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregationcited by 31
  31. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  32. 2023Fully Sparse Fusion for 3D Object Detectioncited by 29
  33. 2023X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detectioncited by 28
  34. 2023UniBEV: Multi-modal 3D Object Detection with Uniform BEV Encoders for Robustness against Missing Sensor Modalitiescited by 28
  35. 2024Is Your HD Map Constructor Reliable under Sensor Corruptions?cited by 28
  36. 2024CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillationcited by 27
  37. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  38. 2023Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusioncited by 25

50 most cited of 194 All 194 (JSON)