mlsanityData to 7 Oct 2026

ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities

arXiv 2407.01525 ↗First posted 1 Jul 2024cs.CV cs.AI cs.CLcited by 47

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Although great progress has been made in 3D visual grounding, current models still rely on explicit textual descriptions for grounding and lack the ability to reason human intentions from implicit instructions. We propose a new task called 3D reasoning grounding and introduce a new benchmark ScanReason which provides over 10K question-answer-location pairs from five reasoning types that require the synerization of reasoning and grounding. We further design our approach, ReGround3D, composed of the visual-centric reasoning module empowered by Multi-modal Large Language Model (MLLM) and the 3D grounding module to obtain accurate object locations by looking back to the enhanced geometry and fine-grained details from the 3D scenes. A chain-of-grounding mechanism is proposed to further boost the performance with interleaved reasoning and grounding steps during inference. Extensive experiments on the proposed benchmark validate the effectiveness of our proposed approach.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 52 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

43 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  2. 2023CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding10 Oct
  3. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
28 more
  1. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  2. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  3. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  4. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  5. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  6. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  7. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  8. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  9. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  10. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  11. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  12. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  13. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  14. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  15. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  16. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  17. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  18. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  19. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  20. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  21. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  22. 2019Linguistic generalization and compositionality in modern artificial neural networkscited by 58
  23. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  24. 20213D Question Answeringcited by 41
  25. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  26. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  27. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  28. 2022Contextual Modeling for 3D Dense Captioning on Point Cloudscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 30; 2026: 9.

47 papers.

Sort:

By area:

  1. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  4. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  5. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  6. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  7. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  8. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  9. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  12. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
35 more
  1. 2025Reasoning in Space via Grounding in the Worldcited by 18
  2. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  3. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  4. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  5. 2025PlaceIt3D: Language-Guided Object Placement in Real 3D Scenescited by 8
  6. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  7. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Modelscited by 7
  8. 2025REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splattingcited by 6
  9. 2025Cook and Clean Together: Teaching Embodied Agents for Parallel Task Executioncited by 6
  10. 2025ProxyTransformation: Preshaping Point Cloud Manifold With Proxy Attention For 3D Visual Groundingcited by 5
  11. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentationcited by 5
  12. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddingscited by 5
  13. 2025VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoningcited by 4
  14. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  15. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  16. 2025Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Groundingcited by 3
  17. 2025MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Imagescited by 3
  18. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  19. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  20. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  21. 2025T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Modelscited by 1
  22. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  23. 2025LLM-RG: Referential Grounding in Outdoor Scenarios using Large Language Modelscited by 1
  24. 2025ChangingGrounding: 3D Visual Grounding in Changing Scenescited by 1
  25. 20263D-DRES: Detailed 3D Referring Expression Segmentationcited by 1
  26. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 1
  27. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Modelscited by 1
  28. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understandingcited by 1
  29. 2025DSM: Constructing a Diverse Semantic Map for 3D Visual Groundingcited by 0
  30. 2025TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraintscited by 0
  31. 2025DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotationcited by 0
  32. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 0
  33. 2026Scaling Diverse Language Generation for 3D Visual Groundingcited by 0
  34. 2026PruneGround: Plug-and-play Spatial Pruning for 3D Visual Groundingcited by 0
  35. 2026S4VY: Segment Anything in Feed-Forward 4D Visual Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Matters for 3D Visual Grounding
  2. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
  3. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
  4. 2025A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding
  5. 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
  6. 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding
  7. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding
  8. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models