mlsanityData to 7 Oct 2026

Multi-modal Situated Reasoning in 3D Scenes

arXiv 2409.02389 ↗First posted 4 Sep 2024cs.CV cs.AI cs.ROcited by 50

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Situation awareness is essential for understanding and reasoning about 3D scenes in embodied AI agents. However, existing datasets and benchmarks for situated understanding are limited in data modality, diversity, scale, and task scope. To address these limitations, we propose Multi-modal Situated Question Answering (MSQA), a large-scale multi-modal situated reasoning dataset, scalably collected leveraging 3D scene graphs and vision-language models (VLMs) across a diverse range of real-world 3D scenes. MSQA includes 251K situated question-answering pairs across 9 distinct question categories, covering complex scenarios within 3D scenes. We introduce a novel interleaved multi-modal input setting in our benchmark to provide text, image, and point cloud for situation and question description, resolving ambiguity in previous single-modality convention (e.g., text). Additionally, we devise the Multi-modal Situated Next-step Navigation (MSNN) benchmark to evaluate models' situated reasoning for navigation. Comprehensive evaluations on MSQA and MSNN highlight the limitations of existing vision-language models and underscore the importance of handling multi-modal interleaved inputs and situation modeling. Experiments on data scaling and cross-domain transfer further demonstrate the efficacy of leveraging MSQA as a pre-training dataset for developing more powerful situated reasoning models.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 75 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

64 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  2. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
38 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2022A Generalist Agentcited by 802
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  9. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  10. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  11. 2023LERF: Language Embedded Radiance Fieldscited by 538
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  13. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  14. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  15. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  16. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  17. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  18. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  19. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  20. 2017Embodied Question Answeringcited by 335
  21. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  22. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  23. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  24. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  25. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  26. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  27. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  28. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  29. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  30. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  31. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  32. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  33. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  34. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  35. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  36. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  37. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  38. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 31; 2026: 12.

50 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  3. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  5. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  6. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  7. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  8. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  9. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 37
  10. 20253D Aware Region Prompted Vision Language Modelcited by 31
  11. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  12. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
38 more
  1. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  2. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  3. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  4. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  5. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  6. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  7. 2024Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Descriptioncited by 9
  8. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  9. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8
  10. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  11. 20253EED: Ground Everything Everywhere in 3Dcited by 8
  12. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  13. 2025Understanding Dynamic Scenes in Ego Centric 4D Point Cloudscited by 6
  14. 2025Hypo3D: Exploring Hypothetical Reasoning in 3Dcited by 5
  15. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  16. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 5
  17. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  18. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 5
  19. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  20. 2025POMA-3D: The Point Map Way to 3D Scene Understandingcited by 3
  21. 20253D Scene Change Modeling With Consistent Multi-View Aggregationcited by 3
  22. 2025Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Modelcited by 2
  23. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  24. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  25. 2025LLaVA3: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMscited by 1
  26. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understandingcited by 1
  27. 2026SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QAcited by 1
  28. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  29. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 1
  30. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
  31. 2026SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Videocited by 1
  32. 2025Error-Driven Scene Editing for 3D Grounding in Large Language Modelscited by 0
  33. 2025SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answeringcited by 0
  34. 2025Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Controlcited by 0
  35. 2026AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifierscited by 0
  36. 2026ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videoscited by 0
  37. 2026Holo-Captioning: Toward the Text Equivalent of 3D Scenescited by 0
  38. 2026Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering
  2. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering
  3. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding
  4. 2026ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA
  5. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs
  6. 2026MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence
  7. 2025Embodied Scene Understanding for Vision Language Models via MetaVQA
  8. 2026CoV: Chain-of-View Prompting for Spatial Reasoning