# VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model arXiv: https://arxiv.org/abs/2603.12655 · HTML: https://mlsanity.com/paper/2603.12655/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in Video-generation world models and simulators, one of 11 directions in World models. - Builds on 49 of its 61 references that are in the map. - First posted on arXiv 13 Mar 2026. ## Placed in - World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/) ## Builds on (most cited first, 30 of 49) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022) - [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023) - [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025) - [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024) - [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024) - [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016) - [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025) - [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023) - [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024) - [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021) - [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024) - [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025) - [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020) - [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024) - [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024) - [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024) - [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025) - [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025) - [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025) - [Autoregressive Video Generation without Vector Quantization](https://mlsanity.com/paper/2412.14169/) (2024) - [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024) ## Built on by (most cited first, 13 of 13) - [SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer](https://mlsanity.com/paper/2605.15178/) (2026) - [Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective](https://mlsanity.com/paper/2604.14025/) (2026) - [PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space](https://mlsanity.com/paper/2607.05373/) (2026) - [GeoWAM: Visual Geometry World Action Models for Autonomous Driving](https://mlsanity.com/paper/2608.23486/) (2026) - [Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3R](https://mlsanity.com/paper/2605.19539/) (2026) - [Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving](https://mlsanity.com/paper/2606.10656/) (2026) - [Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model](https://mlsanity.com/paper/2606.27325/) (2026) - [Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models](https://mlsanity.com/paper/2607.19120/) (2026) - [Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction](https://mlsanity.com/paper/2608.27529/) (2026) - [Geometric 4D Stitching for Grounded 4D Generation](https://mlsanity.com/paper/2605.09984/) (2026) - [GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation](https://mlsanity.com/paper/2609.24981/) (2026) - [PhysWAM: Physically Consistent World Action Model for Autonomous Driving](https://mlsanity.com/paper/2609.37970/) (2026) - [Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models](https://mlsanity.com/paper/2610.01942/) (2026) ## Abstract (arXiv) World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.