mlsanityData to 7 Oct 2026

3D-Aware VLMs with Implicit and Explicit Geometries

arXiv 2607.21595 ↗First posted 23 Jul 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

46 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs19 May
  2. 2026On the Generalization Capacities of MLLMs for Spatial Intelligence5 Mar
  3. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  4. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  5. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
  6. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  7. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  8. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
  9. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  10. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  11. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
1 more
  1. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding18 May

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  10. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
21 more
  1. 2024Depth Anything V2cited by 1,385
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  8. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  9. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  10. 2025AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Viewscited by 234
  11. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  12. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  13. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  14. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  15. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 136
  16. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  17. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  18. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  19. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  20. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  21. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
  2. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  3. 2026Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding
  4. 2026Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
  5. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  6. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations
  7. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models
  8. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models