mlsanityData to 14 Oct 2026

A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving

arXiv 2503.11400 ↗First posted 14 Mar 2025cs.CV cs.ROcited by 5

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) hold the potential to enhance autonomous driving by combining domain-independent world knowledge with context-specific language guidance. Their integration into autonomous driving systems shows promising results in isolated proof-of-concept applications, while their performance is evaluated on selective singular aspects of perception, reasoning, or planning. To leverage their full potential a systematic framework for evaluating MLLMs in the context of autonomous driving is required. This paper proposes a holistic framework for a capability-driven evaluation of MLLMs in autonomous driving. The framework structures scenario understanding along the four core capability dimensions semantic, spatial, temporal, and physical. They are derived from the general requirements of autonomous driving systems, human driver cognition, and language-based reasoning. It further organises the domain into context layers, processing modalities, and downstream tasks such as language-based interaction and decision-making. To illustrate the framework's applicability, two exemplary traffic scenarios are analysed, grounding the proposed dimensions in realistic driving situations. The framework provides a foundation for the structured evaluation of MLLMs' potential for scenario understanding in autonomous driving.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 37 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

27 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement20 Nov
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  3. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  4. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook22 Oct
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 20

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  3. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  4. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  5. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  6. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  7. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  8. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  9. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  10. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 156
  11. 2019The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersectionscited by 137
  12. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
8 more
  1. 2017On a Formal Model of Safe and Scalable Self-driving Carscited by 131
  2. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  3. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  4. 2024Embodied Understanding of Driving Scenarioscited by 63
  5. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  6. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 52
  7. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
  8. 20206-Layer Model for a Structured Description and Categorization of Urban Traffic and Environmentcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  2. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  3. 2025When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTIcited by 0
  4. 2026From Big Data to Fast Data: Towards High-Quality Datasets for Machine Learning Applications from Closed-Loop Data Collectioncited by 0
  5. 2026A Domain-Specific Language for LLM-Driven Trigger Generation in Multimodal Data Collectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing Multimodal LLMs as World Models for Driving
  2. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  3. 2025A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios ↗
  4. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
  5. 2024Application of Multimodal Large Language Models in Autonomous Driving
  6. 2024PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Driving
  7. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving
  8. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles