mlsanityData to 7 Oct 2026

3D-IDE: 3D Implicit Depth Emergent

arXiv 2604.03296 ↗First posted 28 Mar 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Leveraging 3D information within Multimodal Large Language Models (MLLMs) has recently shown significant advantages for indoor scene understanding. However, existing methods, including those using explicit ground-truth 3D positional encoding and those grafting external 3D foundation models for implicit geometry, struggle with the trade-off in 2D-3D representation fusion, leading to suboptimal deployment. To this end, we propose 3D-Implicit Depth Emergence, a method that reframes 3D perception as an emergent property derived from geometric self-supervision rather than explicit encoding. Our core insight is the Implicit Geometric Emergence Principle: by strategically leveraging privileged geometric supervision through mechanisms like a fine-grained geometry validator and global representation constraints, we construct an information bottleneck. This bottleneck forces the model to maximize the mutual information between visual features and 3D structures, allowing 3D awareness to emerge naturally within a unified visual representation. Unlike existing approaches, our method enables 3D perception to emerge implicitly, disentangling features in dense regions and, crucially, eliminating depth and pose dependencies during inference with zero latency overhead. This paradigm shift from external grafting to implicit emergence represents a fundamental rethinking of 3D knowledge integration in visual-language models. Extensive experiments demonstrate that our method surpasses SOTA on multiple 3D scene understanding benchmarks. Our approach achieves a 55% reduction in inference latency while maintaining strong performance across diverse downstream tasks, underscoring the effectiveness of meticulously designed auxiliary objectives for dependency-free 3D understanding. Source code can be found at github.com/ChushanZhang/3D-IDE.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 51 references that are in the map. First posted on arXiv 28 Mar 2026.

Builds on

43 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy29 Sep
  2. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  3. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  4. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning1 Mar
  5. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  6. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
  7. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  8. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  9. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  11. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
22 more
  1. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  2. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  3. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  4. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  5. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  6. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  7. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 176
  8. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  9. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  10. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  11. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  12. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  13. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  14. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  15. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  16. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  17. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  18. 2023NAVI: Category-Agnostic Image Collections with High-Quality 3D Shape and Pose Annotationscited by 53
  19. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  20. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  21. 2025GS-2DGS: Geometrically Supervised 2DGS for Reflective Object Reconstructioncited by 11
  22. 2024DGNS: Deformable Gaussian Splatting and Dynamic Neural Surface for Monocular Dynamic 3D Reconstructioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  2. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  3. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  4. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
  5. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models
  6. 2026Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?
  7. 20263D-Aware VLMs with Implicit and Explicit Geometries
  8. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning