mlsanityData to 7 Oct 2026

End-to-end Autonomous Driving: Challenges and Frontiers

arXiv 2306.16927 ↗First posted 29 Jun 2023cs.RO cs.AI cs.CVcited by 485

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework. we maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving.

Key facts. Cited by 485 papers in the map (data through 7 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 218 of its 320 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

218 of 320 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Hidden Biases of End-to-End Driving Models13 Jun
  2. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes17 May
  3. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving10 May
  4. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives17 Mar
  5. 2022DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles13 Dec
  6. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning15 Jul
  7. 2022MMFN: Multi-Modal-Fusion-Net for End-to-End Driving1 Jul
  8. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline16 Jun
  9. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving31 May
  10. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  11. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observations26 May
  12. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Driving19 Apr
3 more
  1. 2021Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation18 Apr
  2. 2021End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving18 Apr
  3. 2021End-to-end Interpretable Neural Motion Planner17 Jan

Other references · 203

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Segment Anythingcited by 8,158
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  13. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  14. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  15. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  16. 2022Planning-oriented Autonomous Drivingcited by 1,068
  17. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  18. 2016End to End Learning for Self-Driving Carscited by 1,000
  19. 2020Mastering Atari with Discrete World Modelscited by 990
  20. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  21. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  22. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  23. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  24. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  25. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  26. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  27. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  28. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  29. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  30. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  31. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  32. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  33. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  34. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  35. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  36. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  37. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  38. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396

50 most cited of 203 All 203 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 98; 2025: 212; 2026: 151.

485 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  4. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  6. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  7. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  8. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  9. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  10. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  11. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  12. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
38 more
  1. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  2. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  3. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  4. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  5. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
  6. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  7. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  8. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  9. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  10. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  11. 2023Language Prompt for Autonomous Drivingcited by 110
  12. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  13. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  14. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  15. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  16. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  17. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  18. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  19. 2025Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Drivingcited by 84
  20. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  21. 2024Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detectioncited by 66
  22. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  23. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  24. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 52
  25. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  26. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  27. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  28. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  29. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  30. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  31. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 41
  32. 2025iPad: Iterative Proposal-centric End-to-End Autonomous Drivingcited by 41
  33. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  34. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  35. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  36. 2024Learning Manipulation by Predicting Interactioncited by 36
  37. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  38. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34

50 most cited of 485 All 485 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms
  2. 2026Post-Training in End-to-End Autonomous Driving
  3. 2024World Models for Autonomous Driving: An Initial Survey
  4. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation
  5. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions
  6. 2022Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing
  7. 2024Data Scaling Laws for Imitation Learning-Based End-to-End Autonomous Driving
  8. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey