mlsanityData to 14 Oct 2026

VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision

arXiv 2412.14446 ↗First posted 19 Dec 2024cs.CV cs.LGcited by 86

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 2 more
Abstract (arXiv)

Human drivers rely on commonsense reasoning to navigate diverse and dynamic real-world scenarios. Existing end-to-end (E2E) autonomous driving (AD) models are typically optimized to mimic driving patterns observed in data, without capturing the underlying reasoning processes. This limitation constrains their ability to handle challenging driving scenarios. To close this gap, we propose VLM-AD, a method that leverages vision-language models (VLMs) as teachers to enhance training by providing additional supervision that incorporates unstructured reasoning information and structured action labels. Such supervision enhances the model's ability to learn richer feature representations that capture the rationale behind driving patterns. Importantly, our method does not require a VLM during inference, making it practical for real-time deployment. When integrated with state-of-the-art methods, VLM-AD achieves significant improvements in planning accuracy and reduced collision rates on the nuScenes dataset. It further improves route completion and driving scores under closed-loop evaluation, demonstrating its effectiveness in long-horizon, interactive driving scenarios and its potential for safe and reliable real-world deployment.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 66 of its 73 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

66 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles12 Oct
  2. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  3. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  4. 2022Planning-oriented Autonomous Drivingcited by 1,087
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  7. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  8. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  9. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  10. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  11. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  12. 2019Learning by Cheatingcited by 383
  13. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  14. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  15. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  16. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  17. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  18. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  19. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  20. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 248
  21. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  22. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 241
  23. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  24. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  25. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  26. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  27. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 212
  28. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  29. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  30. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  31. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169
  32. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 148
  33. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  34. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 104
  35. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
  36. 2023Hidden Biases of End-to-End Driving Modelscited by 96
  37. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 93
  38. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 89

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 48; 2026: 38.

86 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  2. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  3. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  4. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  5. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  6. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  7. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 44
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  10. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusioncited by 20
  11. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  12. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 17
38 more
  1. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 16
  2. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 16
  3. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  4. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  5. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  6. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 14
  7. 2026AutoDrive-P3: Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuningcited by 14
  8. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 13
  9. 2025VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformerscited by 12
  10. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  11. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 11
  12. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  13. 2025Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 7
  14. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  15. 2025VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splattingcited by 6
  16. 2026StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Drivingcited by 6
  17. 2025DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Drivingcited by 5
  18. 2025VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Drivingcited by 5
  19. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planningcited by 5
  20. 2026Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planningcited by 5
  21. 2026The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Modelscited by 5
  22. 2025VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Drivingcited by 4
  23. 2025CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Drivingcited by 4
  24. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4
  25. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Processcited by 4
  26. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Drivingcited by 4
  27. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Drivingcited by 4
  28. 2025NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Drivingcited by 3
  29. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  30. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  31. 2025iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoningcited by 3
  32. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  33. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  34. 2026ReMoT: Reinforcement Learning with Motion Contrast Tripletscited by 3
  35. 2025Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systemscited by 2
  36. 2025OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Drivingcited by 2
  37. 2025SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Drivingcited by 2
  38. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Drivingcited by 2

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving
  2. 2025AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
  3. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  4. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  5. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
  6. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  7. 2025CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
  8. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving