mlsanityData to 7 Oct 2026

SQA3D: Situated Question Answering in 3D Scenes

arXiv 2210.07474 ↗First posted 14 Oct 2022cs.CV cs.AI cs.CLcited by 303

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment and answer a question under that situation. Based upon 650 scenes from ScanNet, we provide a dataset centered around 6.8k unique situations, along with 20.4k descriptions and 33.4k diverse reasoning questions for these situations. These questions examine a wide spectrum of reasoning capabilities for an intelligent agent, ranging from spatial relation comprehension to commonsense understanding, navigation, and multi-hop reasoning. SQA3D imposes a significant challenge to current multi-modal especially 3D reasoning models. We evaluate various state-of-the-art approaches and find that the best one only achieves an overall score of 47.20%, while amateur human participants can reach 90.06%. We believe SQA3D could facilitate future embodied AI research with stronger situation understanding and reasoning capability.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 57 of its 77 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

57 of 77 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
38 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  13. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  14. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  15. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  16. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  17. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  18. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  19. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  20. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  21. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  22. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  23. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  24. 2017Embodied Question Answeringcited by 335
  25. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  26. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  27. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  28. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  29. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  30. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  31. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
  32. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  33. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  34. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  35. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  36. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  37. 2018PointNetGPD: Detecting Grasp Configurations from Point Setscited by 80
  38. 2022All You May Need for VQA are Image Captionscited by 65

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 28; 2024: 53; 2025: 110; 2026: 111.

303 papers.

Sort:

By area:

  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  2. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  3. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  4. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  5. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  6. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  7. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  8. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  9. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  10. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  11. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  12. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
38 more
  1. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  2. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  3. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  4. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  5. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  6. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  7. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  8. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  9. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  10. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  11. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  12. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  13. 2023An Outlook into the Future of Egocentric Visioncited by 84
  14. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  15. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  16. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  17. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  18. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  19. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  20. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  21. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  22. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  23. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  24. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  25. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  26. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  27. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  28. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  29. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  30. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  31. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  32. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  33. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  34. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  35. 2023Single-view 3D Scene Reconstruction with High-fidelity Shape and Texturecited by 37
  36. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 37
  37. 2023Connecting Multi-modal Contrastive Representationscited by 36
  38. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36

50 most cited of 303 All 303 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering
  2. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering
  3. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering
  4. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understanding
  5. 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
  6. 2026FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy
  7. 2026ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI
  8. 2026Extending Embodied Question Answering from Perception to Decision