mlsanityData to 13 Oct 2026

3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding

arXiv 2401.03201 ↗First posted 6 Jan 2024cs.CV cs.MMcited by 32

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The remarkable potential of multi-modal large language models (MLLMs) in comprehending both vision and language information has been widely acknowledged. However, the scarcity of 3D scenes-language pairs in comparison to their 2D counterparts, coupled with the inadequacy of existing approaches in understanding of 3D scenes by LLMs, poses a significant challenge. In response, we collect and construct an extensive dataset comprising 75K instruction-response pairs tailored for 3D scenes. This dataset addresses tasks related to 3D VQA, 3D grounding, and 3D conversation. To further enhance the integration of 3D spatial information into LLMs, we introduce a novel and efficient prompt tuning paradigm, 3DMIT. This paradigm eliminates the alignment stage between 3D scenes and language and extends the instruction prompt with the 3D modality information including the entire scene and segmented objects. We evaluate the effectiveness of our method across diverse tasks in the 3D scene domain and find that our approach serves as a strategic means to enrich LLMs' comprehension of the 3D world. Our code is available at https://github.com/staymylove/3DMIT.

Key facts. Cited by 32 papers in the map (data through 13 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 21 references that are in the map. First posted on arXiv 6 Jan 2024.

Builds on

17 of 21 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Visual Instruction Tuningcited by 7,827
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,096
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  8. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  9. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 341
  10. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  11. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 234
  12. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
5 more
  1. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  2. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  3. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  4. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 126
  5. 2019VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentationcited by 44

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 11; 2025: 15; 2026: 6.

32 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  3. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  4. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 53
  5. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 42
  7. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  8. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  9. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  10. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  11. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  12. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
20 more
  1. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  2. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  3. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  4. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMscited by 8
  5. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Modelscited by 8
  6. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  7. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  8. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 6
  9. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenescited by 4
  10. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answeringcited by 4
  11. 2025City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learningcited by 2
  12. 2025Omni-SILA: Towards Omni-scene Driven Visual Sentiment Identifying, Locating and Attributing in Videoscited by 1
  13. 2025Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Modelscited by 1
  14. 2026DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoEcited by 1
  15. 2025Handle-based Mesh Deformation Guided By Vision Language Modelcited by 0
  16. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  17. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  18. 2026Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Modelscited by 0
  19. 2026CitySTAR: Structured and Topology-Aware Reasoning for Open-Vocabulary Urban 3D Groundingcited by 0
  20. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
  3. 20243DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset
  4. 2025Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
  5. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
  6. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
  7. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  8. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs