mlsanityData to 7 Oct 2026

A Survey on Multimodal Large Language Models for Autonomous Driving

arXiv 2311.12320 ↗First posted 21 Nov 2023cs.AIcited by 288

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.

Key facts. Cited by 288 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 135 of its 209 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

135 of 209 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct

Other references · 134

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2023Segment Anythingcited by 8,158
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  13. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  14. 2015VQA: Visual Question Answeringcited by 2,489
  15. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  16. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  17. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  18. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  19. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  20. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  21. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  22. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  23. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  24. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  25. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  26. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  27. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  28. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  29. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  30. 2023A Survey on Multimodal Large Language Modelscited by 898
  31. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  32. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  33. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  34. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  35. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  36. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  37. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  38. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657

50 most cited of 134 All 134 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 104; 2025: 127; 2026: 51.

288 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  4. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  5. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  8. 2024ChemLLM: A Chemical Large Language Modelcited by 100
  9. 2025Efficient Reasoning Models: A Surveycited by 93
  10. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  11. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  12. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
38 more
  1. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  2. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  3. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  4. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  5. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  6. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  7. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  9. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  10. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
  11. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  12. 2025Fully Autonomous AI Agents Should Not be Developedcited by 43
  13. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  14. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  15. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  16. 2024Graphic Design with Large Multimodal Modelcited by 38
  17. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  18. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  19. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  20. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  21. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
  22. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  23. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  24. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  25. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  26. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  27. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  28. 2024LLM-Assisted Light: Leveraging Large Language Model Capabilities for Human-Mimetic Traffic Signal Control in Complex Urban Environmentscited by 31
  29. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  30. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  31. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  32. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 30
  33. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  34. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  35. 2024UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarioscited by 28
  36. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  37. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  38. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27

50 most cited of 288 All 288 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  3. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  4. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  5. 2025DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving
  6. 2024Application of Multimodal Large Language Models in Autonomous Driving
  7. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  8. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models