mlsanityData to 7 Oct 2026

NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

arXiv 2305.14836 ↗First posted 24 May 2023cs.CVcited by 258

Not on the map. Why?

Cited by 132 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the autonomous driving scenario, encompassing 34K visual scenes and 460K question-answer pairs. Specifically, we leverage existing 3D detection annotations to generate scene graphs and design question templates manually. Subsequently, the question-answer pairs are generated programmatically based on these templates. Comprehensive statistics prove that our NuScenes-QA is a balanced large-scale benchmark with diverse question formats. Built upon it, we develop a series of baselines that employ advanced 3D detection and VQA techniques. Our extensive experiments highlight the challenges posed by this new task. Codes and dataset are available at https://github.com/qiantianwen/NuScenes-QA.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
25 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  3. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  4. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  5. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  6. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  7. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  8. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  9. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  10. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  11. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  12. 2017Embodied Question Answeringcited by 335
  13. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  14. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  15. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  16. 2018Bilinear Attention Networkscited by 277
  17. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  18. 2023Referring Multi-Object Trackingcited by 91
  19. 2022Anomaly Detection in Autonomous Driving: A Surveycited by 87
  20. 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
  21. 2018Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answeringcited by 51
  22. 20213D Question Answeringcited by 41
  23. 2018Learning to Count Objects in Natural Images for Visual Question Answeringcited by 40
  24. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25
  25. 2022Locate before Answering: Answer Guided Question Localization for Video Question Answeringcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 49; 2025: 109; 2026: 81.

258 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  5. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  7. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  8. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  9. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  10. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  11. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  12. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
38 more
  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  2. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  3. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  4. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  5. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  6. 2023Language Prompt for Autonomous Drivingcited by 110
  7. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  8. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  9. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  10. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  11. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  12. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  13. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  14. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  15. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  16. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  17. 2024Embodied Understanding of Driving Scenarioscited by 63
  18. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  19. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  20. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  21. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  22. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  23. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  24. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  25. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  26. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  27. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  28. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  29. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  30. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  31. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  32. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  33. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  34. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  35. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  36. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  37. 2023NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotationscited by 31
  38. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30

50 most cited of 258 All 258 (JSON)