mlsanityData to 14 Oct 2026

VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction

arXiv 2505.20279 ↗First posted 26 May 2025cs.CV cs.CLcited by 166

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has motivated extending these models to understand 3D scenes, aiming for human-like visual-spatial intelligence. Nevertheless, achieving deep spatial understanding comparable to human capabilities poses significant challenges in model encoding and data acquisition. Existing methods frequently depend on external depth sensors for geometry capture or utilize off-the-shelf algorithms for pre-constructing 3D maps, thereby limiting their scalability, especially with prevalent monocular video inputs and for time-sensitive applications. In this work, we introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding. Leveraging our Spatial-Visual-View Fusion and over 200K curated 3D reconstructive instruction tuning question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial context with language instructions. This enables monocular 3D spatial assistance and embodied reasoning. To facilitate the evaluation of temporal reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark, featuring over 138.6K QA pairs across five distinct tasks focused on evolving spatial relationships. Extensive experiments demonstrate that our model, VLM-3R, not only facilitates robust visual-spatial reasoning but also enables the understanding of temporal 3D context changes, excelling in both accuracy and scalability.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 94 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

66 of 94 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  2. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  5. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  8. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  9. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2024GPT-4o System Cardcited by 4,588
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  13. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  14. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  17. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  18. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  19. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  21. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  22. 2023A Comprehensive Overview of Large Language Modelscited by 648
  23. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  24. 2024Long Context Transfer from Language to Visioncited by 515
  25. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  26. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  27. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  28. 2019Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matchingcited by 417
  29. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  30. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 364
  31. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  32. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  33. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  34. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  35. 20243D Reconstruction with Spatial Memorycited by 277
  36. 2024MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priorscited by 205
  37. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  38. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 42; 2026: 124.

166 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  2. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  3. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  4. 2025Visual Spatial Tuningcited by 89
  5. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  6. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  7. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 46
  8. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 44
  9. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  10. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 38
  11. 2025Human3R: Everyone Everywhere All at Oncecited by 37
  12. 2025From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priorscited by 28
38 more
  1. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  2. 2025SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Modelscited by 26
  3. 2026ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loopcited by 26
  4. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 25
  5. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  6. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 23
  7. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  8. 2025AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backendcited by 20
  9. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 19
  10. 2025Reasoning in Space via Grounding in the Worldcited by 19
  11. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 18
  12. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 17
  13. 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoningcited by 17
  14. 2026ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoningcited by 15
  15. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 14
  16. 2025Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Taskscited by 13
  17. 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videoscited by 13
  18. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  19. 2025Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Modelscited by 12
  20. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 12
  21. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  22. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  23. 2026Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Trainingcited by 11
  24. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  25. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  26. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 8
  27. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  28. 2026Make Geometry Matter for Spatial Reasoningcited by 8
  29. 2025Generalizable Operating Room Expert with Multimodal Enhancementcited by 7
  30. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  31. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  32. 2025Video Spatial Reasoning with Object-Centric 3D Rolloutcited by 6
  33. 2026Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoningcited by 6
  34. 2026OpenSpatial: A Principled Data Engine for Empowering Spatial Intelligencecited by 6
  35. 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspectivecited by 6
  36. 2026GEM: Generative Supervision Helps Embodied Intelligencecited by 6
  37. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
  38. 2025Vision-Language Memory for Spatial Reasoningcited by 5

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  2. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  3. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
  4. 2024I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction
  5. 2025Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy
  6. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
  7. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
  8. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision