Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering
Not on the map. Why?
Abstract (arXiv)
3D Scene Question Answering (3D SQA) represents an interdisciplinary task that integrates 3D visual perception and natural language processing, empowering intelligent agents to comprehend and interact with complex 3D environments. Recent advances in large multimodal modelling have driven the creation of diverse datasets and spurred the development of instruction-tuning and zero-shot methods for 3D SQA. However, this rapid progress introduces challenges, particularly in achieving unified analysis and comparison across datasets and baselines. In this survey, we provide the first comprehensive and systematic review of 3D SQA. We organize existing work from three perspectives: datasets, methodologies, and evaluation metrics. Beyond basic categorization, we identify shared architectural patterns across methods. Our survey further synthesizes core limitations and discusses how current trends, such as instruction tuning, multimodal alignment, and zero-shot, can shape future developments. Finally, we propose a range of promising research directions covering dataset construction, task generalization, interaction modeling, and unified evaluation protocols. This work aims to serve as a foundation for future research and foster progress toward more generalizable and intelligent 3D SQA systems.
Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 75 of its 112 references that are in the map. First posted on arXiv 1 Feb 2025.
Builds on
75 of 112 references.
References · 75
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2017Mask R-CNNcited by 7,259
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
38 more
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2015VQA: Visual Question Answeringcited by 2,489
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2022Language-driven Semantic Segmentationcited by 711
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
- 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
- 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
- 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
- 2016Semantic Scene Completion from a Single Depth Imagecited by 468
- 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
- 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
- 2023An Embodied Generalist Agent in 3D Worldcited by 355
- 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
- 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
- 2017Embodied Question Answeringcited by 335
- 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
- 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
- 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
- 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
- 2021Group-Free 3D Object Detection via Transformerscited by 215
- 2021Stochastic Scene-Aware Motion Predictioncited by 203
- 2023Scalable 3D Captioning with Pretrained Modelscited by 201
- 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
- 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
- 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
- 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
- 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
50 most cited of 75 All 75 (JSON)
Built on by
Citing papers per year: 2025: 6; 2026: 5.
11 papers.
- 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
- 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
- 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 4
- 2025Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptionscited by 3
- 2025POMA-3D: The Point Map Way to 3D Scene Understandingcited by 3
- 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answeringcited by 2
- 2025RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusioncited by 1
- 2025DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Taskscited by 0
- 2026RGB-Pointmap Pretraining for Unified 3D Scene Understandingcited by 0
- 2026IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenancecited by 0
- 2026An overview of 3D Vision-Language Modelscited by 0
- 2026An overview of 3D Vision-Language Models4 Sep 2026
- 2026IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance25 Apr 2026
- 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments15 Apr 2026
- 2026RGB-Pointmap Pretraining for Unified 3D Scene Understanding2 Apr 2026
- 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering31 Mar 2026
- 2025RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusion20 Dec 2025
- 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning9 Dec 2025
- 2025POMA-3D: The Point Map Way to 3D Scene Understanding20 Nov 2025
- 2025DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks21 Aug 2025
- 2025Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions19 Jul 2025
- 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts27 May 2025
Similar, not cited
Close in wording, no citation either way
- 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering
- 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding
- 2026ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA
- 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering
- 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering
- 2026CoV: Chain-of-View Prompting for Spatial Reasoning
- 2025SpatialReasoner: Active Perception for Large-Scale 3D Scene Understanding
- 2024Agent3D-Zero: An Agent for Zero-shot 3D Understanding