mlsanityData to 7 Oct 2026

Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model

arXiv 2405.17427 ↗First posted 27 May 2024cs.CVcited by 27

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in multimodal large language models (LLMs) have demonstrated significant potential across various domains, particularly in concept reasoning. However, their applications in understanding 3D environments remain limited, primarily offering textual or numerical outputs without generating dense, informative segmentation masks. This paper introduces Reason3D, a novel LLM designed for comprehensive 3D understanding. Reason3D processes point cloud data and text prompts to produce textual responses and segmentation masks, enabling advanced tasks such as 3D reasoning segmentation, hierarchical searching, express referring, and question answering with detailed mask outputs. We propose a hierarchical mask decoder that employs a coarse-to-fine approach to segment objects within expansive scenes. It begins with a coarse location estimation, followed by object mask estimation, using two unique tokens predicted by LLMs based on the textual query. Experimental results on large-scale ScanNet and Matterport3D datasets validate the effectiveness of our Reason3D across various tasks.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 65 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

52 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2020Point Transformercited by 1,462
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  7. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  11. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  12. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  13. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  14. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  15. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  16. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  17. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  18. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  19. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  20. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  21. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  22. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  23. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  24. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  25. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  26. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  27. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  28. 2021Hierarchical Aggregation for 3D Instance Segmentationcited by 126
  29. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  30. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  31. 2022Superpoint Transformer for 3D Scene Instance Segmentationcited by 115
  32. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  33. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  34. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  35. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  36. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  37. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  38. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 10.

27 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  3. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  4. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  5. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  6. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  7. 2025Multimodal Referring Segmentation: A Surveycited by 22
  8. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  9. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  10. 2025Conditional Panoramic Image Generation via Masked Autoregressive Modelingcited by 9
  11. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  12. 2025PlaceIt3D: Language-Guided Object Placement in Real 3D Scenescited by 8
15 more
  1. 2026MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentationcited by 8
  2. 2025REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splattingcited by 6
  3. 2025Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Groundercited by 2
  4. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  5. 2026Segment and Select: Vision-Language Segmentation in 3D Scenarioscited by 2
  6. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understandingcited by 2
  7. 20263D Segmentation Using Viewpoint-Dependent Spatial Relationshipscited by 1
  8. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understandingcited by 1
  9. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  10. 2025DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotationcited by 0
  11. 2026Direction-aware 3D Large Multimodal Modelscited by 0
  12. 2026Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentationcited by 0
  13. 2026VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detectioncited by 0
  14. 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Modelscited by 0
  15. 2026Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models
  2. 2025OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model
  3. 2024PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model
  4. 2025Text-to-Scene with Large Reasoning Models
  5. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation
  6. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  7. 20233D-LLM: Injecting the 3D World into Large Language Models
  8. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models