mlsanityData to 7 Oct 2026

LingoQA: Visual Question Answering for Autonomous Driving

arXiv 2312.14115 ↗First posted 21 Dec 2023cs.RO cs.AI cs.CVcited by 144

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

We introduce LingoQA, a novel dataset and benchmark for visual question answering in autonomous driving. The dataset contains 28K unique short video scenarios, and 419K annotations. Evaluating state-of-the-art vision-language models on our benchmark shows that their performance is below human capabilities, with GPT-4V responding truthfully to 59.6% of the questions compared to 96.6% for humans. For evaluation, we propose a truthfulness classifier, called Lingo-Judge, that achieves a 0.95 Spearman correlation coefficient to human evaluations, surpassing existing techniques like METEOR, BLEU, CIDEr, and GPT-4. We establish a baseline vision-language model and run extensive ablation studies to understand its performance. We release our dataset and benchmark as an evaluation platform for vision-language models in autonomous driving.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 54 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

37 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving9 Nov
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
23 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  2. 2019Attention is not Explanationcited by 916
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  6. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  8. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  11. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  12. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  13. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  14. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  15. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  16. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  17. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  18. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  19. 2017Interpretable Learning for Self-Driving Cars by Visualizing Causal Attentioncited by 106
  20. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  21. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  22. 2022From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answeringcited by 51
  23. 2021Reimagining an autonomous vehiclecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 60; 2026: 66.

144 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  3. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  4. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  5. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  7. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  8. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  9. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 52
  10. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  11. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 40
  12. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
38 more
  1. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  2. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  3. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  4. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  5. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  6. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 33
  7. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  8. 2024WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Modelcited by 28
  9. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
  10. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 26
  11. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  12. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Taskscited by 23
  13. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Drivingcited by 23
  14. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  15. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  16. 2025StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Drivingcited by 20
  17. 2025VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understandingcited by 20
  18. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  19. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  20. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  21. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  22. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spacescited by 17
  23. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  24. 2025LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Modelscited by 16
  25. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  26. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 14
  27. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  28. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  29. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  30. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  31. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  32. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  33. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  34. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  35. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Drivingcited by 9
  36. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Modelscited by 8
  37. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  38. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Drivingcited by 8

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations
  2. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving
  3. 2026Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Driving
  4. 2026Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression
  5. 2025TinyDrive: Multiscale Visual Question Answering with Selective Token Routing for Autonomous Driving
  6. 2025Unlock the Power of Unlabeled Data in Language Driving Model
  7. 2023DriveLM: Driving with Graph Visual Question Answering
  8. 2025RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System