mlsanityData to 7 Oct 2026

NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations

arXiv 2312.06352 ↗First posted 11 Dec 2023cs.CV cs.CLcited by 31

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Visual Question Answering (VQA) is one of the most important tasks in autonomous driving, which requires accurate recognition and complex situation evaluations. However, datasets annotated in a QA format, which guarantees precise language generation and scene recognition from driving scenes, have not been established yet. In this work, we introduce Markup-QA, a novel dataset annotation technique in which QAs are enclosed within markups. This approach facilitates the simultaneous evaluation of a model's capabilities in sentence generation and VQA. Moreover, using this annotation methodology, we designed the NuScenes-MQA dataset. This dataset empowers the development of vision language models, especially for autonomous driving tasks, by focusing on both descriptive capabilities and precise QA. The dataset is available at https://github.com/turingmotors/NuScenes-MQA.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 33 of its 41 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

33 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving11 Sep

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
20 more
  1. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  3. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  4. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  5. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  7. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  8. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  9. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  10. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  11. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  12. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  13. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  14. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  15. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  16. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  17. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  18. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  19. 2018Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answeringcited by 51
  20. 2023PALI: A Language Identification Benchmark for Perso-Arabic Scriptscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 12.

31 papers.

Sort:

By area:

  1. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  2. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  3. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  5. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  6. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  7. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervisioncited by 16
  8. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 14
  9. 2025RefAV: Towards Planning-Centric Scenario Miningcited by 13
  10. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  11. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  12. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
19 more
  1. 2025V3LMA: Visual 3D-enhanced Language Model for Autonomous Drivingcited by 7
  2. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  3. 2024Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patternscited by 4
  4. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  5. 2025TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videoscited by 2
  6. 2025MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understandingcited by 2
  7. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  8. 2025nuScenes Revisited: Progress and Challenges in Autonomous Drivingcited by 2
  9. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Drivingcited by 2
  10. 2024Precise Drive with VLM: First Prize Solution for PRCV 2024 Drive LM challengecited by 1
  11. 2025Temporal Object Captioning for Street Scene Videos from LiDAR Trackscited by 1
  12. 2026Wild-Drive: Off-Road Scene Captioning and Path Planning via Robust Multi-modal Routing and Efficient Large Language Modelcited by 0
  13. 2026DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understandingcited by 0
  14. 2026C2T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordinationcited by 0
  15. 2026SceneMiner: Identity-Preserving Multi-Task Fine-Tuning for Unified BEV Scene Miningcited by 0
  16. 2026Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 0
  17. 2026VOLA: Improving Open-World Driving by VLM-Based Semantic Attribute Predictioncited by 0
  18. 2026Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Drivingcited by 0
  19. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LingoQA: Visual Question Answering for Autonomous Driving
  2. 2025Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
  3. 2025Robust Driving QA through Metadata-Grounded Context and Task-Specific Prompts
  4. 2025TinyDrive: Multiscale Visual Question Answering with Selective Token Routing for Autonomous Driving
  5. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving
  6. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  7. 2026Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression
  8. 2026Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Driving