mlsanityData to 14 Oct 2026

VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model

arXiv 2603.12655 ↗First posted 13 Mar 2026cs.CVcited by 13

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 49 of its 61 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

49 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Back to the Features: DINO as a Foundation for Video World Models25 Jul
  2. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling10 Jul
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr

Other references · 44

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2022Video Diffusion Modelscited by 2,305
  7. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  8. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  9. 2024Depth Anything V2cited by 1,422
  10. 2025DINOv3cited by 1,414
  11. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  12. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
32 more
  1. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  4. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  5. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  6. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  7. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 506
  8. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  9. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  10. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  11. 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
  12. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
  13. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  14. 2024Autoregressive Video Generation without Vector Quantizationcited by 186
  15. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
  16. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  17. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  18. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  19. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
  20. 2025Aether: Geometric-Aware Unified World Modelingcited by 97
  21. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
  22. 2025Latent Diffusion Model without Variational Autoencodercited by 88
  23. 2024World-consistent Video Diffusion with Explicit 3D Modelingcited by 65
  24. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 45
  25. 2024DINO-Foresight: Looking into the Future with DINOcited by 45
  26. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 37
  27. 2026Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generationcited by 36
  28. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 31
  29. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 19
  30. 2026MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sourcescited by 10
  31. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
  32. 2025GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026GeoWM: Efficient Direct World Modeling in Explicit Geometry
  2. 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modeling
  3. 2025VFMF: World Modeling by Forecasting Vision Foundation Model Features
  4. 2026Flow Matching in Feature Space for Stochastic World Modeling
  5. 2025Video World Models with Long-term Spatial Memory
  6. 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
  7. 2026World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
  8. 2026Envisioning the Future, One Step at a Time