mlsanityData to 7 Oct 2026

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

arXiv 2104.09224 ↗First posted 19 Apr 2021cs.CV cs.AI cs.LGcited by 372

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

How should representations from complementary sensors be integrated for autonomous driving? Geometry-based sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based on existing sensor fusion methods under-perform in the presence of a high density of dynamic agents and complex scenarios, which require global contextual reasoning, such as handling traffic oncoming from multiple directions at uncontrolled intersections. Therefore, we propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.

Key facts. Cited by 372 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 52 of its 68 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

52 of 68 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  7. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  8. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  11. 2019Learning by Cheatingcited by 361
  12. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
38 more
  1. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
  2. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  3. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 296
  4. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  5. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 228
  6. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  7. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  8. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  9. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  10. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  11. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  12. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 147
  13. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  14. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  15. 2020Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?cited by 139
  16. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  17. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
  18. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  19. 2019Depth Completion from Sparse LiDAR Data with Depth-Normal Constraintscited by 108
  20. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  21. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  22. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  23. 2019Does computer vision matter for action?cited by 64
  24. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  25. 2018Semi-parametric Image Synthesiscited by 50
  26. 2019Sensor Fusion for Joint 3D Object Detection and Semantic Segmentationcited by 48
  27. 2020Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Drivingcited by 41
  28. 2019Monocular Plan View Networks for Autonomous Drivingcited by 37
  29. 2017CAR-Net: Clairvoyant Attentive Recurrent Networkcited by 34
  30. 2020Label Efficient Visual Abstractions for Autonomous Drivingcited by 33
  31. 2020MultiXNet: Multiclass Multistage Multimodal Motion Predictioncited by 31
  32. 2020Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Drivingcited by 24
  33. 2019Looking to Relations for Future Trajectory Forecastcited by 23
  34. 2020LaserFlow: Efficient and Probabilistic Object Detection and Motion Forecastingcited by 22
  35. 2020STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Predictioncited by 22
  36. 2020Depth Completion via Inductive Fusion of Planar LIDAR and Monocular Cameracited by 22
  37. 2020MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Viewscited by 20
  38. 2017Brain Inspired Cognitive Model with Attention for Self-Driving Carscited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 66; 2023: 79; 2024: 62; 2025: 84; 2026: 63.

372 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  4. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  5. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  6. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  10. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  11. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  12. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
38 more
  1. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  2. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  3. 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 243
  4. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  5. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  6. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  7. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  8. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  9. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  10. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  11. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  12. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  13. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  14. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  15. 2023V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perceptioncited by 155
  16. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  17. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  18. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  19. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  20. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  21. 2022Learning from All Vehiclescited by 135
  22. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  23. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  24. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  25. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  26. 2021A Survey of Visual Transformerscited by 117
  27. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 115
  28. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  29. 2021Learning to drive from a world on railscited by 114
  30. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  31. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  32. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  33. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  34. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  35. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 95
  36. 2022COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehiclescited by 95
  37. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  38. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 93

50 most cited of 372 All 372 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
  2. 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving
  3. 2020Probabilistic End-to-End Vehicle Navigation in Complex Dynamic Environments with Multimodal Sensor Fusion
  4. 2026MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning
  5. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations
  6. 2024DeepInteraction++: Multi-Modality Interaction for Autonomous Driving
  7. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale
  8. 2025MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer