mlsanityData to 14 Oct 2026

Application of Multimodal Large Language Models in Autonomous Driving

arXiv 2412.16410 ↗First posted 21 Dec 2024cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

In this era of technological advancements, several cutting-edge techniques are being implemented to enhance Autonomous Driving (AD) systems, focusing on improving safety, efficiency, and adaptability in complex driving environments. However, AD still faces some problems including performance limitations. To address this problem, we conducted an in-depth study on implementing the Multi-modal Large Language Model. We constructed a Virtual Question Answering (VQA) dataset to fine-tune the model and address problems with the poor performance of MLLM on AD. We then break down the AD decision-making process by scene understanding, prediction, and decision-making. Chain of Thought has been used to make the decision more perfectly. Our experiments and detailed analysis of Autonomous Driving give an idea of how important MLLM is for AD.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 38 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

25 of 38 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Segment Anythingcited by 8,234
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  10. 2022Planning-oriented Autonomous Drivingcited by 1,087
  11. 2023A Survey on Multimodal Large Language Modelscited by 909
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
13 more
  1. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  2. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  3. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  4. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  5. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  6. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  7. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  8. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  9. 2020Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologiescited by 41
  10. 2023Dynamically Conservative Self-Driving Planner for Long-Tail Casescited by 13
  11. 2022Multi-Task Conditional Imitation Learning for Autonomous Navigation at Crowded Intersectionscited by 4
  12. 2024Enhancing Bangla Language Next Word Prediction and Sentence Completion through Extended RNN with Bi-LSTM Model On N-gram Languagecited by 3
  13. 2024Sentiment Polarity Analysis of Bangla Food Reviews Using Machine and Deep Learning Algorithmscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  2. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  3. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  4. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  6. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  7. 2024Probing Multimodal LLMs as World Models for Driving
  8. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving