mlsanityData to 14 Oct 2026

TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving

arXiv 2205.15997 ↗First posted 31 May 2022cs.CV cs.AI cs.LGcited by 485

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g. object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.

Key facts. Cited by 485 papers in the map (data through 14 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 104 of its 134 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

104 of 134 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policies28 Sep
  2. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Driving19 Apr
  3. 2021Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles16 Apr

Other references · 101

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  8. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  9. 2019Objects as Pointscited by 1,175
  10. 2020Designing Network Design Spacescited by 960
  11. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  12. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
38 more
  1. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  2. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  3. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  4. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  5. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  6. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  7. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 397
  8. 2019Learning by Cheatingcited by 383
  9. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  10. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 341
  11. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 331
  12. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 326
  13. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 298
  14. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 278
  15. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  16. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  17. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 229
  18. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  19. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  20. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 194
  21. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  22. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  23. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 179
  24. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  25. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  26. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  27. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  28. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  29. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 156
  30. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 153
  31. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 149
  32. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 148
  33. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  34. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 144
  35. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  36. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 139
  37. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  38. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 60; 2024: 70; 2025: 152; 2026: 194.

485 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,087
  2. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  4. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  6. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  7. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  8. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  9. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  10. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 252
  11. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 249
  12. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 242
38 more
  1. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  2. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  3. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 212
  4. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
  5. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  6. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169
  7. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  8. 2023A Language Agent for Autonomous Drivingcited by 164
  9. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 150
  10. 2025Pseudo-Simulation for Autonomous Drivingcited by 150
  11. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 144
  12. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 141
  13. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  14. 2025Unified Vision-Language-Action Modelcited by 134
  15. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 126
  16. 2025GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Drivingcited by 120
  17. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  18. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
  19. 2023Hidden Biases of End-to-End Driving Modelscited by 96
  20. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  21. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 96
  22. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 94
  23. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 93
  24. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  25. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 90
  26. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  27. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 86
  28. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 86
  29. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  30. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 79
  31. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  32. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  33. 2025Robust Autonomy Emerges from Self-Playcited by 67
  34. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 67
  35. 2025Generalized Trajectory Scoring for End-to-end Multimodal Planningcited by 67
  36. 2025DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Drivingcited by 64
  37. 2025A Survey of World Models for Autonomous Drivingcited by 60
  38. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 58

50 most cited of 485 All 485 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Cognitive TransFuser: Semantics-guided Transformer-based Sensor Fusion for Improved Waypoint Prediction
  2. 2024MaskFuser: Masked Fusion of Joint Multi-Modal Tokenization for End-to-End Autonomous Driving
  3. 2024Crossfusor: A Cross-Attention Transformer Enhanced Conditional Diffusion Model for Car-Following Trajectory Prediction
  4. 2020Probabilistic End-to-End Vehicle Navigation in Complex Dynamic Environments with Multimodal Sensor Fusion
  5. 2021OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication
  6. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey
  7. 2026PlanTRansformer: Unified Prediction and Planning with Goal-conditioned Transformer
  8. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization