mlsanityData to 7 Oct 2026

VLP: Vision Language Planning for Autonomous Driving

arXiv 2401.05577 ↗First posted 10 Jan 2024cs.CVcited by 125

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

Autonomous driving is a complex and challenging task that aims at safe motion planning through scene understanding and reasoning. While vision-only autonomous driving methods have recently achieved notable performance, through enhanced scene understanding, several key issues, including lack of reasoning, low generalization performance and long-tail scenarios, still need to be addressed. In this paper, we present VLP, a novel Vision-Language-Planning framework that exploits language models to bridge the gap between linguistic understanding and autonomous driving. VLP enhances autonomous driving systems by strengthening both the source memory foundation and the self-driving car's contextual understanding. VLP achieves state-of-the-art end-to-end planning performance on the challenging NuScenes dataset by achieving 35.9% and 60.5% reduction in terms of average L2 error and collision rates, respectively, compared to the previous best method. Moreover, VLP shows improved performance in challenging long-tail scenarios and strong generalization capabilities when faced with new urban environments.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 34 of its 36 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

34 of 36 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
22 more
  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2016End to End Learning for Self-Driving Carscited by 1,000
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  6. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  7. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  8. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  9. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  10. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  11. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  12. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  13. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  14. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  15. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  16. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90
  17. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 83
  18. 2021Monocular Quasi-Dense 3D Object Trackingcited by 80
  19. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 49
  20. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  21. 2022Towards Long-Tailed 3D Detectioncited by 40
  22. 2023Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videoscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 68; 2026: 39.

125 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  3. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  4. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  5. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  6. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  7. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  8. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  9. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  10. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  11. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  12. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
38 more
  1. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  2. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  3. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  4. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  5. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  6. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  7. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  8. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  9. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  10. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  11. 2025RaceVLA: VLA-based Racing Drone Navigation with Human-like Behaviourcited by 22
  12. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  13. 2025CognitiveDrone: A VLA Model and Evaluation Benchmark for Real-Time Cognitive Task Solving and Reasoning in UAVscited by 21
  14. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  15. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  16. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  17. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  18. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validationcited by 15
  19. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  20. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 15
  21. 2024CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flowcited by 14
  22. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  23. 2024LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Drivingcited by 12
  24. 2025VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformerscited by 12
  25. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  26. 2025OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Frameworkcited by 11
  27. 2025Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansioncited by 11
  28. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  29. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  30. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  31. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  32. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  33. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  34. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 10
  35. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  36. 2025ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Modelscited by 9
  37. 2025Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competitioncited by 9
  38. 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 9

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
  3. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  4. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  5. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  6. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
  7. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving
  8. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving