mlsanityData to 7 Oct 2026

TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers

arXiv 2203.11496 ↗First posted 22 Mar 2022cs.CVcited by 464

On the map: World models → World modeling from LiDAR generation

Abstract (arXiv)

LiDAR and camera are two important sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, the robustness against inferior image conditions, e.g., bad illumination and sensor misalignment, is under-explored. Existing fusion methods are easily affected by such conditions, mainly due to a hard association of LiDAR points and image pixels, established by calibration matrices. We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. Specifically, our TransFusion consists of convolutional backbones and a detection head based on a transformer decoder. The first layer of the decoder predicts initial bounding boxes from a LiDAR point cloud using a sparse set of object queries, and its second decoder layer adaptively fuses the object queries with useful image features, leveraging both spatial and contextual relationships. The attention mechanism of the transformer enables our model to adaptively determine where and what information should be taken from the image, leading to a robust and effective fusion strategy. We additionally design an image-guided query initialization strategy to deal with objects that are difficult to detect in point clouds. TransFusion achieves state-of-the-art performance on large-scale datasets. We provide extensive experiments to demonstrate its robustness against degenerated image quality and calibration errors. We also extend the proposed method to the 3D tracking task and achieve the 1st place in the leaderboard of nuScenes tracking, showing its effectiveness and generalization capability.

Key facts. Cited by 464 papers in the map (data through 7 Oct 2026). Placed in World modeling from LiDAR generation, one of 11 directions in World models. Builds on 62 of its 74 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

62 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation2 Dec
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks4 Jun

Other references · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  6. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  7. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  8. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  9. 2019Objects as Pointscited by 1,172
  10. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  11. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  12. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
38 more
  1. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  2. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  3. 2017Focal Loss for Dense Object Detectioncited by 749
  4. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  5. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  6. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  7. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  8. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  9. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  10. 2017Deep Layer Aggregationcited by 436
  11. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  12. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  13. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  14. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  15. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  16. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  17. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  18. 2021Voxel Transformer for 3D Object Detectioncited by 247
  19. 2021Group-Free 3D Object Detection via Transformerscited by 215
  20. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  21. 2018Orthographic Feature Transform for Monocular 3D Object Detectioncited by 201
  22. 20203D Object Detection with Pointformercited by 194
  23. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  24. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  25. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  26. 2021Multimodal Virtual Point 3D Detectioncited by 166
  27. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  28. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 162
  29. 2021RangeDet:In Defense of Range View for LiDAR-based 3D Object Detectioncited by 140
  30. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  31. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  32. 2019IoU Loss for 2D/3D Object Detectioncited by 128
  33. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  34. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  35. 2021RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detectioncited by 115
  36. 2021Improving 3D Object Detection with Channel-wise Transformercited by 108
  37. 2021FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detectioncited by 92
  38. 2019Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspotscited by 84

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 47; 2023: 130; 2024: 121; 2025: 96; 2026: 70.

464 papers.

Sort:

By area:

  1. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  5. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  6. 2023VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Trackingcited by 221
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  8. 2022DETRs with Hybrid Matchingcited by 149
  9. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  10. 2022Vision-Centric BEV Perception: A Surveycited by 126
  11. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  12. 2023Spherical Transformer for LiDAR-based 3D Recognitioncited by 124
38 more
  1. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  2. 2023DSVT: Dynamic Sparse Voxel Transformer with Rotated Setscited by 122
  3. 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
  4. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  5. 2023FB-BEV: BEV Representation from Forward-Backward View Transformationscited by 98
  6. 2024An Extensible Framework for Open Heterogeneous Collaborative Perceptioncited by 87
  7. 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
  8. 2023Virtual Sparse Convolution for Multimodal 3D Object Detectioncited by 81
  9. 2023CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perceptioncited by 81
  10. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  11. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  12. 2023RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Drivingcited by 76
  13. 2023ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detectioncited by 76
  14. 2022LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perceptioncited by 74
  15. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  16. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  17. 2023FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformercited by 72
  18. 2022LargeKernel3D: Scaling up Kernels in 3D Sparse CNNscited by 71
  19. 2023HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformercited by 70
  20. 2023Sparse4D v3: Advancing End-to-End 3D Detection and Trackingcited by 69
  21. 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 68
  22. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  23. 2022CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformercited by 66
  24. 2024TUMTraf V2X Cooperative Perception Datasetcited by 66
  25. 2024Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detectioncited by 66
  26. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  27. 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
  28. 2023FocalFormer3D : Focusing on Hard Instance for 3D Object Detectioncited by 58
  29. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  30. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  31. 2022Fully Convolutional One-Stage 3D Object Detection on LiDAR Range Imagescited by 56
  32. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  33. 2023PillarNeXt: Rethinking Network Designs for 3D Object Detection in LiDAR Point Cloudscited by 54
  34. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 52
  35. 2024SAFDNet: A Simple and Effective Network for Fully Sparse 3D Object Detectioncited by 52
  36. 2023PointOcc: Cylindrical Tri-Perspective View for Point-based 3D Semantic Occupancy Predictioncited by 51
  37. 2022CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusioncited by 50
  38. 2022Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detectioncited by 49

50 most cited of 464 All 464 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Resilient Sensor Fusion under Adverse Sensor Failures via Multi-Modal Expert Fusion
  2. 2022MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer for Autonomous Driving
  3. 20263DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds
  4. 2023Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
  5. 2021LiDAR Cluster First and Camera Inference Later: A New Perspective Towards Autonomous Driving
  6. 2024V2X-R: Cooperative LiDAR-4D Radar Fusion with Denoising Diffusion for 3D Object Detection
  7. 2024LiDAR-PTQ: Post-Training Quantization for Point Cloud 3D Object Detection
  8. 2025Diffusion-Based Restoration for Multi-Modal 3D Object Detection in Adverse Weather