VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.
Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 49 of its 61 references that are in the map. First posted on arXiv 13 Mar 2026.
Builds on
49 of 61 references.
Closest ancestors · 5
Same direction, newest first
- 2025Back to the Features: DINO as a Foundation for Video World Models25 Jul
- 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling10 Jul
- 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
- 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
Other references · 44
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
- 2022Video Diffusion Modelscited by 2,305
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
- 2024Depth Anything V2cited by 1,422
- 2025DINOv3cited by 1,414
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2024Grounding Image Matching in 3D with MASt3Rcited by 983
32 more
- 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
- 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
- 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
- 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
- 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
- 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 506
- 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
- 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
- 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
- 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
- 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
- 2025World Simulation with Video Foundation Models for Physical AIcited by 200
- 2024Autoregressive Video Generation without Vector Quantizationcited by 186
- 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
- 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
- 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
- 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
- 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
- 2025Aether: Geometric-Aware Unified World Modelingcited by 97
- 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
- 2025Latent Diffusion Model without Variational Autoencodercited by 88
- 2024World-consistent Video Diffusion with Explicit 3D Modelingcited by 65
- 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 45
- 2024DINO-Foresight: Looking into the Future with DINOcited by 45
- 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 37
- 2026Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generationcited by 36
- 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 31
- 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 19
- 2026MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sourcescited by 10
- 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
- 2025GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generationcited by 3
Built on by
Citing papers per year: 2026: 13.
13 papers.
Sort:
By area:
- 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 55
- 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspectivecited by 6
- 2026PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Spacecited by 4
- 2026GeoWAM: Visual Geometry World Action Models for Autonomous Drivingcited by 3
- 2026Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3Rcited by 2
- 2026Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Drivingcited by 1
- 2026Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Modelcited by 1
- 2026Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Modelscited by 1
- 2026Revisiting Local Context for Long-Horizon Streaming 3D Reconstructioncited by 1
- 2026Geometric 4D Stitching for Grounded 4D Generationcited by 0
- 2026GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generationcited by 0
- 2026PhysWAM: Physically Consistent World Action Model for Autonomous Drivingcited by 0
- 2026Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models1 Oct 2026
- 2026PhysWAM: Physically Consistent World Action Model for Autonomous Driving29 Sep 2026
- 2026GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation21 Sep 2026
- 2026Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction27 Aug 2026
- 2026GeoWAM: Visual Geometry World Action Models for Autonomous Driving24 Aug 2026
- 2026Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models21 Jul 2026
- 2026PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space6 Jul 2026
- 2026Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model25 Jun 2026
- 2026Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving9 Jun 2026
- 2026Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3R19 May 2026
- 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer14 May 2026
- 2026Geometric 4D Stitching for Grounded 4D Generation11 May 2026
1 more
- 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective15 Apr 2026
- 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 55
- 2026GeoWAM: Visual Geometry World Action Models for Autonomous Drivingcited by 3
- 2026Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Modelcited by 1
- 2026PhysWAM: Physically Consistent World Action Model for Autonomous Drivingcited by 0
- 2026Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Modelscited by 0
- 2026Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models1 Oct 2026
- 2026PhysWAM: Physically Consistent World Action Model for Autonomous Driving29 Sep 2026
- 2026GeoWAM: Visual Geometry World Action Models for Autonomous Driving24 Aug 2026
- 2026Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model25 Jun 2026
- 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer14 May 2026
- 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspectivecited by 6
- 2026Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3Rcited by 2
- 2026Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Drivingcited by 1
- 2026Revisiting Local Context for Long-Horizon Streaming 3D Reconstructioncited by 1
- 2026Geometric 4D Stitching for Grounded 4D Generationcited by 0
- 2026GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generationcited by 0
- 2026GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation21 Sep 2026
- 2026Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction27 Aug 2026
- 2026Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving9 Jun 2026
- 2026Trust It or Not: Evidential Uncertainty for Feed-Forward 3D Reconstruction with Trust3R19 May 2026
- 2026Geometric 4D Stitching for Grounded 4D Generation11 May 2026
- 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective15 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026GeoWM: Efficient Direct World Modeling in Explicit Geometry
- 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modeling
- 2025VFMF: World Modeling by Forecasting Vision Foundation Model Features
- 2026Flow Matching in Feature Space for Stochastic World Modeling
- 2025Video World Models with Long-term Spatial Memory
- 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
- 2026World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
- 2026Envisioning the Future, One Step at a Time