mlsanityData to 14 Oct 2026

Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving

arXiv 2503.21505 ↗First posted 27 Mar 2025cs.CL cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Existing benchmarks for Vision-Language Model (VLM) on autonomous driving (AD) primarily assess interpretability through open-form visual question answering (QA) within coarse-grained tasks, which remain insufficient to assess capabilities in complex driving scenarios. To this end, we introduce VLADBench, a challenging and fine-grained dataset featuring close-form QAs that progress from static foundational knowledge and elements to advanced reasoning for dynamic on-road situations. The elaborate VLADBench spans 5 key domains: Traffic Knowledge Understanding, General Element Recognition, Traffic Graph Generation, Target Attribute Comprehension, and Ego Decision-Making and Planning. These domains are further broken down into 11 secondary aspects and 29 tertiary tasks for a granular evaluation. A thorough assessment of general and domain-specific (DS) VLMs on this benchmark reveals both their strengths and critical limitations in AD contexts. To further exploit the cognitive and reasoning interactions among the 5 domains for AD understanding, we start from a small-scale VLM and train the DS models on individual domain datasets (collected from 1.4M DS QAs across public sources). The experimental results demonstrate that the proposed benchmark provides a crucial step toward a more comprehensive assessment of VLMs in AD, paving the way for the development of more cognitively sophisticated and reasoning-capable AD systems.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 83 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

68 of 83 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives7 Jan
  2. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving10 Dec
  3. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving29 Oct
  4. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving4 Sep
  5. 2024SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving31 Jul
  6. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases16 Apr
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  8. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  9. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  10. 2023LingoQA: Visual Question Answering for Autonomous Driving21 Dec
  11. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec
  12. 2023Dolphins: Multimodal Language Model for Driving1 Dec
1 more
  1. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2022Planning-oriented Autonomous Drivingcited by 1,087
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  11. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  13. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  15. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  16. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  17. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  18. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  19. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  20. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  21. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  22. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  23. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  24. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  25. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  26. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  27. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  28. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  29. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  30. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  31. 2023Language-Driven Representation Learning for Roboticscited by 165
  32. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  33. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  34. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  35. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  36. 2023OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mappingcited by 83
  37. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  38. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learningcited by 38
  2. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  3. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  4. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Drivingcited by 9
  5. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 9
  6. 2025RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Drivingcited by 8
  7. 2025MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusioncited by 3
  8. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  9. 2025E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Drivingcited by 2
  10. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2
  11. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  12. 2026Towards Safer Mobile Agents: Scalable Generation and Evaluation of Diverse Scenarios for VLMscited by 1
7 more
  1. 2026From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Drivingcited by 1
  2. 2026The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Modelscited by 1
  3. 2025RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understandingcited by 0
  4. 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?cited by 0
  5. 2026Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectificationcited by 0
  6. 2026Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Modelscited by 0
  7. 2026DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Executioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLMs Guided Interpretable Decision Making for Autonomous Driving
  2. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models
  3. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
  4. 2025AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Driving
  5. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
  6. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving
  7. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving
  8. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving