mlsanityData to 7 Oct 2026

Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning

arXiv 2503.00513 ↗First posted 1 Mar 2025cs.CVcited by 50

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Despite encouraging progress in 3D scene understanding, it remains challenging to develop an effective Large Multi-modal Model (LMM) that is capable of understanding and reasoning in complex 3D environments. Most previous methods typically encode 3D point and 2D image features separately, neglecting interactions between 2D semantics and 3D object properties, as well as the spatial relationships within the 3D environment. This limitation not only hinders comprehensive representations of 3D scene, but also compromises training and inference efficiency. To address these challenges, we propose a unified Instance-aware 3D Large Multi-modal Model (Inst3D-LMM) to deal with multiple 3D scene understanding tasks simultaneously. To obtain the fine-grained instance-level visual tokens, we first introduce a novel Multi-view Cross-Modal Fusion (MCMF) module to inject the multi-view 2D semantics into their corresponding 3D geometric features. For scene-level relation-aware tokens, we further present a 3D Instance Spatial Relation (3D-ISR) module to capture the intricate pairwise spatial relationships among objects. Additionally, we perform end-to-end multi-task instruction tuning simultaneously without the subsequent task-specific fine-tuning. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods across 3D scene understanding, reasoning and grounding tasks. Source code is available at https://github.com/hanxunyu/Inst3D-LMM

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 65 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

52 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  2. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding6 Jan
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  4. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes12 Apr

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
36 more
  1. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  2. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  3. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  4. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  5. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  6. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  7. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  8. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  9. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  10. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  11. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  12. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  13. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  14. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  15. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  16. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  17. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  18. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  19. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  20. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  21. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  22. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  23. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  24. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  25. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  26. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 66
  27. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  28. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  29. 2023CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detectioncited by 51
  30. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  31. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  32. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  33. 2019VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentationcited by 44
  34. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  35. 20233DRP-Net: 3D Relative Position-aware Network for 3D Visual Groundingcited by 25
  36. 2024LLMI3D: MLLM-based 3D Perception from a Single 2D Imagecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 29.

50 papers.

Sort:

By area:

  1. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  2. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  3. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  4. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  5. 2025Reasoning in Space via Grounding in the Worldcited by 18
  6. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  7. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  8. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  9. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  10. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 12
  11. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 9
  12. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
38 more
  1. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  2. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 6
  3. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  4. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  5. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  6. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangementcited by 3
  7. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splattingcited by 3
  8. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  9. 2025CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3Dcited by 2
  10. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  11. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  12. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  13. 2026Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMscited by 2
  14. 2026Unlocking Dense Metric Depth Estimation in VLMscited by 2
  15. 2026Unified 3D Scene Understanding Through Physical World Modelingcited by 2
  16. 2026Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMscited by 2
  17. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understandingcited by 1
  18. 2025A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Groundingcited by 1
  19. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 1
  20. 20263D-IDE: 3D Implicit Depth Emergentcited by 1
  21. 2025DSM: Constructing a Diverse Semantic Map for 3D Visual Groundingcited by 0
  22. 2025Error-Driven Scene Editing for 3D Grounding in Large Language Modelscited by 0
  23. 2025Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referringcited by 0
  24. 2026On the Adversarial Robustness of 3D Large Vision-Language Modelscited by 0
  25. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMscited by 0
  26. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  27. 2026MAG-3D: Multi-Agent Grounded Reasoning for 3D Understandingcited by 0
  28. 2026PLAF: Pixel-wise Language-Aligned Feature Extraction for Efficient 3D Scene Understandingcited by 0
  29. 2026Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignmentcited by 0
  30. 2026Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priorscited by 0
  31. 2026CAPruner: Conceptual-Adjacent Scene Graph Pruner for Enhancing 3D Spatial Reasoning of Large Language Modelscited by 0
  32. 2026Scaling Diverse Language Generation for 3D Visual Groundingcited by 0
  33. 2026CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Modelscited by 0
  34. 2026Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understandingcited by 0
  35. 2026Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Modelscited by 0
  36. 2026CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMscited by 0
  37. 2026From Alignment to Fusion in 3D Vision-Languagecited by 0
  38. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2025Reg3D: Reconstructive Geometry Instruction Tuning for 3D Scene Understanding
  3. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  4. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
  5. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMs
  6. 2026Beyond Single Object: Learning 3D Relations with Large Language Models
  7. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  8. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts