mlsanityData to 14 Oct 2026

Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

arXiv 2603.19235 ↗First posted 19 Mar 2026cs.CV cs.ROcited by 7

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 96 of its 105 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

96 of 105 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning9 Dec
  2. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling1 Dec
  3. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views21 Oct
  4. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  5. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence29 May
  6. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoning2 Apr
  7. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Training1 Apr
  8. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models27 Mar
  9. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
  10. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
1 more
  1. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct

Other references · 83

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  11. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
38 more
  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  4. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  6. 2025DINOv3cited by 1,414
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  8. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  12. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  13. 2024Long Context Transfer from Language to Visioncited by 515
  14. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  15. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  16. 2025VACE: All-in-One Video Creation and Editingcited by 479
  17. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  18. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  19. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  20. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  21. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  22. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  23. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  24. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  25. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  26. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  27. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  28. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  29. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  30. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  31. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  32. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 182
  33. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
  34. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  35. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  36. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  37. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  38. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models
  2. 2024Compositional 3D-aware Video Generation with LLM Director
  3. 2026Gen4U: Unifying Video Generation and Understanding via Diffusion
  4. 2026PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generation
  5. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
  6. 2025Video4Spatial: Towards Visuospatial Intelligence with Context-Guided Video Generation
  7. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction
  8. 2026SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space