mlsanityData to 7 Oct 2026

ScanQA: 3D Question Answering for Spatial Scene Understanding

arXiv 2112.10482 ↗First posted 20 Dec 2021cs.CVcited by 340

Not on the map. Why?

Cited by 135 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a new 3D spatial understanding task of 3D Question Answering (3D-QA). In the 3D-QA task, models receive visual information from the entire 3D scene of the rich RGB-D indoor scan and answer the given textual questions about the 3D scene. Unlike the 2D-question answering of VQA, the conventional 2D-QA models suffer from problems with spatial understanding of object alignment and directions and fail the object identification from the textual questions in 3D-QA. We propose a baseline model for 3D-QA, named ScanQA model, where the model learns a fused descriptor from 3D object proposals and encoded sentence embeddings. This learned descriptor correlates the language expressions with the underlying geometric features of the 3D scan and facilitates the regression of 3D bounding boxes to determine described objects in textual questions and outputs correct answers. We collected human-edited question-answer pairs with free-form answers that are grounded to 3D objects in each 3D scene. Our new ScanQA dataset contains over 40K question-answer pairs from the 800 indoor scenes drawn from the ScanNet dataset. To the best of our knowledge, the proposed 3D-QA task is the first large-scale effort to perform object-grounded question-answering in 3D environments.

Key facts. Cited by 340 papers in the map (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
29 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  13. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  14. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  15. 2017Embodied Question Answeringcited by 335
  16. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  17. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  18. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  19. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  20. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  21. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  22. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  23. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  24. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  25. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  26. 2019Multi-Target Embodied Question Answeringcited by 74
  27. 2020Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rulecited by 22
  28. 2020Visual Question Answering on 360deg Imagescited by 15
  29. 2020Simplifying Paragraph-level Question Generation via Transformer Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 33; 2024: 65; 2025: 121; 2026: 110.

340 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  3. 2023LERF: Language Embedded Radiance Fieldscited by 538
  4. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  5. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  8. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  9. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  10. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  11. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  12. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
38 more
  1. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  2. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  3. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  4. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  5. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  6. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  7. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  8. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  9. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  10. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  11. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  12. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  13. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  14. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  15. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  16. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  17. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  18. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  19. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  20. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  21. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  22. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  23. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  24. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  25. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  26. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  27. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  28. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  29. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  30. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  31. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  32. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  33. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  34. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  35. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  36. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  37. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  38. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50

50 most cited of 340 All 340 (JSON)