mlsanityData to 7 Oct 2026

Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases

arXiv 2404.10595 ↗First posted 16 Apr 2024cs.CVcited by 75

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have received widespread attention for advancing the interpretable self-driving. Existing evaluations of LVLMs primarily focus on multi-faceted capabilities in natural circumstances, lacking automated and quantifiable assessment for self-driving, let alone the severe road corner cases. In this work, we propose CODA-LM, the very first benchmark for the automatic evaluation of LVLMs for self-driving corner cases. We adopt a hierarchical data structure and prompt powerful LVLMs to analyze complex driving scenes and generate high-quality pre-annotations for the human annotators, while for LVLM evaluation, we show that using the text-only large language models (LLMs) as judges reveals even better alignment with human preferences than the LVLM judges. Moreover, with our CODA-LM, we build CODA-VLM, a new driving LVLM surpassing all open-sourced counterparts on CODA-LM. Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task. We hope CODA-LM can become the catalyst to promote interpretable self-driving empowered by LVLMs.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 61 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

36 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  2. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving9 Nov

Other references · 34

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2022Planning-oriented Autonomous Drivingcited by 1,068
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
22 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  3. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  4. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  5. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  6. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  7. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 176
  8. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  9. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  10. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  11. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  12. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  13. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  14. 2021MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Drivingcited by 41
  15. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  16. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  17. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  18. 2021Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approachescited by 27
  19. 2022Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routingcited by 18
  20. 2024Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Expertscited by 16
  21. 2023InfMLLM: A Unified Framework for Visual-Language Taskscited by 11
  22. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 34; 2026: 21.

75 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  4. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  5. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  6. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  7. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  8. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  9. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  10. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  11. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  12. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
38 more
  1. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  2. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22
  3. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  4. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  5. 2024AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Drivingcited by 14
  6. 2025SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generationcited by 14
  7. 2024MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Drivingcited by 13
  8. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  9. 2024Generating Out-Of-Distribution Scenarios Using Language Modelscited by 10
  10. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  11. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  12. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  13. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Drivingcited by 9
  14. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  15. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  16. 2024RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Modelscited by 7
  17. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Modelscited by 7
  18. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7
  19. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 6
  20. 2024Edge-Cloud Collaborative Motion Planning for Autonomous Driving with Large Language Modelscited by 5
  21. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  22. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  23. 2025ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Drivingcited by 5
  24. 2024PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Drivingcited by 4
  25. 2025Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Drivingcited by 4
  26. 2025DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Modelscited by 4
  27. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Modelscited by 4
  28. 2025RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Casecited by 4
  29. 2025MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machinescited by 3
  30. 2025iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoningcited by 3
  31. 2025Universal Camouflage Attack on Vision-Language Models for Autonomous Drivingcited by 3
  32. 2025DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Riskscited by 3
  33. 2025RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation Systemcited by 3
  34. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  35. 2026A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous drivingcited by 3
  36. 2025AutoDrive-QA: A Multiple-Choice Benchmark for Vision-Language Evaluation in Urban Autonomous Drivingcited by 2
  37. 2025Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systemscited by 2
  38. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  3. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  4. 2026V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  6. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
  7. 2026ScenePilot-4K: A Large-Scale First-Person Dataset and Benchmark for Vision-Language Models in Autonomous Driving
  8. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving