# Can Video World Models Track Unobserved World States? arXiv: https://arxiv.org/abs/2608.30692 ยท HTML: https://mlsanity.com/paper/2608.30692/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Video-generation world models and simulators, one of 11 directions in World models. - Builds on 55 of its 85 references that are in the map. - First posted on arXiv 31 Aug 2026. ## Placed in - World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/) ## Builds on (most cited first, 30 of 55) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023) - [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024) - [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025) - [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024) - [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024) - [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018) - [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025) - [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024) - [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026) - [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025) - [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024) - [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024) - [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024) - [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026) - [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022) - [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025) - [Looped Transformers as Programmable Computers](https://mlsanity.com/paper/2301.13196/) (2023) - [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024) - [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025) - [The Illusion of State in State-Space Models](https://mlsanity.com/paper/2404.08819/) (2024) - [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025) - [Flex Attention: A Programming Model for Generating Optimized Attention Kernels](https://mlsanity.com/paper/2412.05496/) (2024) - [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025) - [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025) - [Video World Models with Long-term Spatial Memory](https://mlsanity.com/paper/2506.05284/) (2025) - [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026) ## Built on by (most cited first, 2 of 2) - [Scaling Video Generation for Reasoning: At What Cost?](https://mlsanity.com/paper/2609.36599/) (2026) - [Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object](https://mlsanity.com/paper/2610.07355/) (2026) ## Abstract (arXiv) Video world models are increasingly used as simulators, but visual fidelity alone does not show that a model maintains the hidden state of the world. We examine this difference with an action-conditioned video Shell Game, a visual analogue of S5 state tracking that separates visual rendering from compositing the unobserved world state. Trained on 5-swap chains, standard backbones (e.g., bidirectional and autoregressive Transformers, Mamba, and linear attention) render plausible videos and predict the correct ball location up to 5 swaps. However, they fail to learn the rule and generalize to longer swap chains, even with more denoising steps. As the pixel-based diffusion loss does not force the generated frames to hold the unseen ball position, output tokens cannot carry it, and the state has to live within the architecture. In a causal Transformer, this implicit state is an append-only KV cache, which is written once and never revised, so the model must re-compose the swaps at every chunk. Tracking S5 this way requires depth to grow with sequence length, which no fixed-depth Transformer provides. We study what enables learning the rule, and find that length generalization requires a revisable state carried across chunks and an update expressive enough to apply a swap. Linear attention can achieve this by allowing negative transition eigenvalues, and autoregressive Transformers can do so with nonlinear TTT fast weights (e.g., SwiGLU) whose online updates change the feature map used to read their state. We further examine Memory Maze and Block World, where the state is not fixed by the input action stream alone and must be corrected from observations or keeps changing out of view, and discuss the implications for building stateful video world models.