mlsanityData to 14 Oct 2026

Vision Language Models in Autonomous Driving: A Survey and Outlook

arXiv 2310.14414 ↗First posted 22 Oct 2023cs.CV cs.AIcited by 126

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

The applications of Vision-Language Models (VLMs) in the field of Autonomous Driving (AD) have attracted widespread attention due to their outstanding performance and the ability to leverage Large Language Models (LLMs). By incorporating language data, driving systems can gain a better understanding of real-world environments, thereby enhancing driving safety and efficiency. In this work, we present a comprehensive and systematic survey of the advances in vision language models in this domain, encompassing perception and understanding, navigation and planning, decision-making and control, end-to-end autonomous driving, and data generation. We introduce the mainstream VLM tasks in AD and the commonly utilized metrics. Additionally, we review current studies and applications in various areas and summarize the existing language-enhanced autonomous driving datasets thoroughly. Lastly, we discuss the benefits and challenges of VLMs in AD and provide researchers with the current research gaps and future trends.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 124 of its 173 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

124 of 173 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 122

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  12. 2023Visual Instruction Tuningcited by 7,869
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,414
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2023A Survey of Large Language Modelscited by 2,576
  13. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  14. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  15. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  16. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  17. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  18. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  19. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  20. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  21. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  22. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  23. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  24. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  25. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  26. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  27. 2022Planning-oriented Autonomous Drivingcited by 1,087
  28. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  29. 2016End to End Learning for Self-Driving Carscited by 1,009
  30. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  31. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  32. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  33. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  34. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  35. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  36. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  37. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  38. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 69; 2026: 46.

126 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 176
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 39
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  6. 2024ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObilitycited by 34
  7. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  8. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  9. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  10. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advancescited by 21
  11. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  12. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
38 more
  1. 2025A Survey on Efficient Vision-Language Modelscited by 19
  2. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 18
  3. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  4. 2025O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Modelscited by 15
  5. 2025InstructPart: Task-Oriented Part Segmentation with Instruction Reasoningcited by 14
  6. 2025Trajectory Prediction Meets Large Language Models: A Surveycited by 14
  7. 2024ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understandingcited by 13
  8. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 13
  9. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 12
  10. 2025Traffic-R1: Reinforced LLMs Bring Human-Like Reasoning to Traffic Signal Control Systemscited by 12
  11. 2024Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objectscited by 11
  12. 2025Towards Human-Centric Autonomous Driving: A Fast-Slow Architecture Integrating Large Language Model Guidance with Reinforcement Learningcited by 11
  13. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understandingcited by 11
  14. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  15. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  16. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 10
  17. 2024Large-scale moral machine experiment on large language modelscited by 9
  18. 2024World Models: The Safety Perspectivecited by 9
  19. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  20. 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarioscited by 7
  21. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  22. 2026Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Drivingcited by 7
  23. 2025LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Taskscited by 6
  24. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  25. 2025NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Drivingcited by 6
  26. 2025RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policiescited by 6
  27. 2024Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Surveycited by 5
  28. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  29. 2024From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Drivingcited by 5
  30. 2025INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluationcited by 5
  31. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Drivingcited by 5
  32. 2025Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networkscited by 5
  33. 2025Automated Vehicles Should be Connected with Natural Languagecited by 5
  34. 2025Pedestrian Intention Prediction via Vision-Language Foundation Modelscited by 5
  35. 2025Survey of GenAI for Automotive Software Development: From Requirements to Executable Codecited by 5
  36. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  37. 2025A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routingcited by 5
  38. 2025Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Drivingcited by 4

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  2. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
  3. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  5. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
  6. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
  7. 2026Towards Reliable Vision-Language Models for Autonomous Driving
  8. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models