mlsanityData to 7 Oct 2026

Driving VLMs and open-vocabulary detection

390 papers1 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: driving, autonomous, autonomous driving, detection, vlms

Key facts. Driving VLMs and open-vocabulary detection: 390 papers on arXiv since 2021 (data through 7 Oct 2026). 92 of its 390 mapped papers (24%) cite “DriveLM: Driving with Graph Visual Question Answering” (2023, arXiv 2312.14150). 82 of its 390 mapped papers (21%) cite “DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models” (2024, arXiv 2402.12289). 11 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

11 new papers, 17 Sep – 7 Oct; 1 this week.

  1. This week · 1
  2. Not All Uncertainty Matters: Simulation-in-the-Loop Fast-Slow Reasoning for Decision-Critical Autonomous Driving System
  3. Earlier · 10
  4. Localized Conformal Safety Monitoring with Vision-Language Models for Autonomous Driving
  5. Towards Reliable Vision-Language Models for Autonomous Driving
  6. Front-to-Back: Benchmarking Vision-Language Models for Asymmetric Cross-View Vehicle Re-Identification
  7. DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
  8. Vision-Language-Action Autonomous Driving Agent with Language-based Memory
  9. Speed in the Blind Spot: An Interpretability Analysis of Dynamic Perception in VLMs for Autonomous Driving
  10. AnchorReasoning: A Visual Grounding and Causal Reasoning Dataset in Long-Tail Autonomous Driving Scenarios
  11. CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning
  12. Towards Systematic Qualification of Vision-Language Models for Automotive Perception Systems
All 11
  1. VeriFuse: Bounded Vision-Language Arbitration and Reason-Guided Refinement for Cooperative 3D Perception also fits: Embodied and 3D vision-language models

Foundations

Most cited by papers here

  1. DriveLM: Driving with Graph Visual Question Answering2023cited by 92 here
  2. DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models2024cited by 82 here
  3. DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2023cited by 81 here
  4. DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving2023cited by 40 here
  5. LingoQA: Visual Question Answering for Autonomous Driving2023cited by 36 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 24 here
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 36 here
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 16 here
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 26 here
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 82 here
  6. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 26 here
18 more it builds on, not in this chain
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 15 here
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 81 here
  3. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 20 here
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 29 here
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 30 here
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 27 here
  7. 2023Dolphins: Multimodal Language Model for Drivingcited by 30 here
  8. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 32 here
  9. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 40 here
  10. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 36 here
  11. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 92 here
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 29 here
  13. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 26 here
  14. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 17 here
  15. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 18 here
  16. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 20 here
  17. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 30 here
  18. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 27 here

The 24 most cited of 675.

Browse

30 of 390, ranked 6–35

  1. 6Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving202332
  2. 7Vision Language Models in Autonomous Driving: A Survey and Outlook202330
  3. 8Dolphins: Multimodal Language Model for Driving202330
  4. 9Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives202527
  5. 10A Survey on Multimodal Large Language Models for Autonomous Driving202327
  6. 11Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models202426
  7. 12Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving202426
  8. 13OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning202420
  9. 14Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving202320
  10. 15Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases202418
All 30
  1. 16Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving202417
  2. 17DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding202515
  3. 18OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving202413
  4. 19VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision202412
  5. 20OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning202511
  6. 21VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving202411
  7. 22Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving202510
  8. 23ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving202510
  9. 24RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model202410
  10. 25Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model202210
  11. 26NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models20259
  12. 27HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models20239
  13. 28AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving20258
  14. 29Scaling Open-Vocabulary Object Detection20238
  15. 30On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving20238
  16. 31When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis20257
  17. 32NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving20257
  18. 33STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving20257
  19. 34A Survey on Vision-Language-Action Models for Autonomous Driving20257
  20. 35WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios20257