# A Control Theory of Predictability in Latent World Models

arXiv: https://arxiv.org/abs/2607.10362 · HTML: https://mlsanity.com/paper/2607.10362/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 11 Jul 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 15 of 15)

- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics](https://mlsanity.com/paper/2511.08544/) (2025)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Linearly-Recurrent Autoencoder Networks for Learning Dynamics](https://mlsanity.com/paper/1712.01378/) (2017)
- [VAMPnets: Deep learning of molecular kinetics](https://mlsanity.com/paper/1710.06012/) (2017)
- [Variational approach for learning Markov processes from time series data](https://mlsanity.com/paper/1707.04659/) (2017)
- [Rigorous data-driven computation of spectral properties of Koopman operators for dynamical systems](https://mlsanity.com/paper/2111.14889/) (2021)
- [Residual Dynamic Mode Decomposition: Robust and verified Koopmanism](https://mlsanity.com/paper/2205.09779/) (2022)
- [When Does LeJEPA Learn a World Model?](https://mlsanity.com/paper/2605.26379/) (2026)

## Built on by (most cited first, 9 of 9)

- [What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations](https://mlsanity.com/paper/2607.27017/) (2026)
- [Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency](https://mlsanity.com/paper/2608.12939/) (2026)
- [Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation](https://mlsanity.com/paper/2609.10954/) (2026)
- [Finite-Sample Metric Non-Collapse for Geometrically Supervised Latent World Models in Control](https://mlsanity.com/paper/2608.07265/) (2026)
- [Beyond Task Success: Stage-Wise Reliability of World Model Planning under Sensing Degradation](https://mlsanity.com/paper/2609.07126/) (2026)
- [D-JEPA: A Decision-Aligned Latent World Model](https://mlsanity.com/paper/2609.24749/) (2026)
- [Adaptive Latent Capacity for World Models](https://mlsanity.com/paper/2609.32921/) (2026)
- [Beyond a single latent space: a dual-latent world model for long-horizon planning](https://mlsanity.com/paper/2609.37644/) (2026)
- [When Low Prediction Error Misleads Planning: Diagnosing Representation, Dynamics, and Decision Failures in Latent World Models](https://mlsanity.com/paper/2610.05550/) (2026)

## Abstract (arXiv)

Latent world models are trained to predict future states in a learned representation and are then deployed inside a planner that selects actions by simulating them forward. Current practice adopts the prediction error, the single- or multi-step rollout loss on held-out data, as the training and model-selection objective, on the assumption that a lower prediction error yields better control. We show that this assumption is unreliable for a structural reason: a planner does not query the model on the training distribution but on the states that its candidate actions reach, which generally leave the data manifold, so an error averaged over the data cannot by itself govern control. We therefore reframe the objective as the discrepancy between the predicted and the true plan-cost at the plan the planner commits to, and prove that the planner's suboptimality is bounded by twice this discrepancy, whereas the data-averaged prediction error neither bounds nor tracks it. Under a linear-control premise the discrepancy separates into two terms. The first is a small on-manifold residual, on which the predicted and true dynamics agree and which a spectral tax prices through the non-normality of the latent transition operator. The second is an off-manifold divergence, on which an action carries the state off the manifold and the two dynamics diverge; this divergence is the binding term and is bounded by no data-averaged error. Synthetic operators confirm the pricing formulas, and latent model-predictive control experiments confirm the decoupling: across seeds, the single-step validation error is essentially uncorrelated with control success, whereas a fidelity score on the planner-reachable measure tracks it.