# Latent State Design for World Models under Sufficiency Constraints

arXiv: https://arxiv.org/abs/2605.01694 · HTML: https://mlsanity.com/paper/2605.01694/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 50 of its 61 references that are in the map.
- First posted on arXiv 3 May 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 30 of 50)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2503.20523/) (2025)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [Video World Models with Long-term Spatial Memory](https://mlsanity.com/paper/2506.05284/) (2025)
- [Scalable methods for computing state similarity in deterministic Markov Decision Processes](https://mlsanity.com/paper/1911.09291/) (2019)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [RELIC: Interactive Video World Model with Long-Horizon Memory](https://mlsanity.com/paper/2512.04040/) (2025)
- [The Value Equivalence Principle for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2011.03506/) (2020)
- [Unsupervised Predictive Memory in a Goal-Directed Agent](https://mlsanity.com/paper/1803.10760/) (2018)
- [Temporal Predictive Coding For Model-Based Planning In Latent Space](https://mlsanity.com/paper/2106.07156/) (2021)

## Built on by (most cited first, 3 of 3)

- [Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Models](https://mlsanity.com/paper/2608.15156/) (2026)
- [What a World Model Represents Is Three Questions](https://mlsanity.com/paper/2607.06640/) (2026)
- [Quantum-Structured World Models (QSWMs) for Predictive Latent Dynamics](https://mlsanity.com/paper/2608.05371/) (2026)

## Abstract (arXiv)

A world model matters to an agent only through the state it constructs. That state must preserve some information, discard other information, and support some future function: prediction, control, planning, memory, grounding, or counterfactual reasoning. This paper treats world-model research as latent state design under sufficiency constraints. We propose a functional taxonomy that groups methods by what their latent state is for, rather than by architecture or application domain: predictive embedding, recurrent belief state, object/causal structure, latent action interface, grounded planning interface, and memory substrate. These roles expose distinctions that architecture-based groupings hide, including the gap between predictive sufficiency and control sufficiency, and the gap between passive video prediction and counterfactual action modeling. The taxonomy supports an evaluation framework that judges a model by the sufficiency constraint its latent state was built to satisfy. We compare methods along seven axes: representation, prediction, planning, controllability, causal/counterfactual support, memory, and uncertainty. We use the resulting matrix as a diagnostic for what a latent state preserves, discards, and enables. The conclusion that follows is that an actionable world model is the one whose state construction matches the task, not the one that preserves the most information.