mlsanityData to 14 Oct 2026

Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors

arXiv 2505.24625 ↗First posted 30 May 2025cs.CV cs.AIcited by 120

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 68 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

59 of 68 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  2. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  3. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
  4. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  5. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  6. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models10 Dec
  7. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  8. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  9. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model1 Aug
  10. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  11. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  12. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
2 more
  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  11. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
33 more
  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  3. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  5. 2024Yi: Open Foundation Models by 01.AIcited by 719
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  8. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  9. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  10. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  11. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  13. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  14. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  15. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  16. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  17. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  18. 2022Visual Spatial Reasoningcited by 293
  19. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 286
  20. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  21. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  22. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  23. 2016Single-Image Depth Perception in the Wildcited by 180
  24. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  25. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  26. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 141
  27. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  28. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  29. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  30. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  31. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  32. 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
  33. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 95.

120 papers.

Sort:

By area:

  1. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  2. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 29
  5. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 28
  6. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  7. 2026VGGT-Omegacited by 26
  8. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
  9. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 19
  10. 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Modelcited by 18
  11. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 18
  12. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 17
38 more
  1. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  2. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  3. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  4. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  5. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  6. 2026Make Geometry Matter for Spatial Reasoningcited by 8
  7. 2026On the Generalization Capacities of MLLMs for Spatial Intelligencecited by 7
  8. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  9. 2025Understanding Dynamic Scenes in Ego Centric 4D Point Cloudscited by 6
  10. 2025Video Spatial Reasoning with Object-Centric 3D Rolloutcited by 6
  11. 2026Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoningcited by 6
  12. 2026Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligencecited by 6
  13. 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspectivecited by 6
  14. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  15. 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
  16. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  17. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidancecited by 5
  18. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  19. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representationscited by 5
  20. 2026Cambrian-P: Pose-Grounded Video Understandingcited by 5
  21. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understandingcited by 5
  22. 2025Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergycited by 4
  23. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understandingcited by 4
  24. 2026Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMscited by 4
  25. 2026Geometry-Guided 3D Visual Token Pruning for Video-Language Modelscited by 4
  26. 20264DThinker: Thinking with 4D Imagery for Dynamic Spatial Understandingcited by 4
  27. 2026ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Modelscited by 4
  28. 2026Unlocking Dense Metric Depth Estimation in VLMscited by 4
  29. 2026Understanding the Impact of Geometric Foundation Models on Vision-Language-Action Modelscited by 4
  30. 2026Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streamscited by 4
  31. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  32. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 3
  33. 2026Geometry-Guided Camera Motion Understanding in VideoLLMscited by 3
  34. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Modelscited by 3
  35. 2026Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Mapscited by 3
  36. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  37. 2026Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scalecited by 3
  38. 2026CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmarkcited by 3

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
  3. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
  4. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM
  5. 2026Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning
  6. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  7. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
  8. 20233D-LLM: Injecting the 3D World into Large Language Models