# World models of environment, agent and joint agent-environment systems

arXiv: https://arxiv.org/abs/2608.20401 · HTML: https://mlsanity.com/paper/2608.20401/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 22 of 22)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [A Theory of Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2203.00397/) (2022)
- [Social World Models](https://mlsanity.com/paper/2509.00559/) (2025)
- [Constrained belief updates explain geometric structures in transformer representations](https://mlsanity.com/paper/2502.01954/) (2025)
- [Plasticity as the Mirror of Empowerment](https://mlsanity.com/paper/2505.10361/) (2025)
- [NeuroAI for AI Safety](https://mlsanity.com/paper/2411.18526/) (2024)
- [Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning](https://mlsanity.com/paper/2511.22226/) (2025)
- [Reinforcement Learning: An Overview](https://mlsanity.com/paper/2412.05265/) (2024)
- [Transformers learn factored representations](https://mlsanity.com/paper/2602.02385/) (2026)
- [Neural networks leverage nominally quantum and post-quantum representations](https://mlsanity.com/paper/2507.07432/) (2025)
- [A Bayesian Interpretation of the Internal Model Principle](https://mlsanity.com/paper/2503.00511/) (2025)
- [Next-token pretraining implies in-context learning](https://mlsanity.com/paper/2505.18373/) (2025)
- [On Kalman-Bucy filters, linear quadratic control and active inference](https://mlsanity.com/paper/2005.06269/) (2020)
- [From monoliths to modules: Decomposing transducers for efficient world modelling](https://mlsanity.com/paper/2512.02193/) (2025)
- [Information Thermodynamics of Agents: The Work Capacity of Channels with Memory](https://mlsanity.com/paper/2504.06209/) (2025)
- [Neural Computers](https://mlsanity.com/paper/2604.06425/) (2026)
- [The World Is Bigger! A Computationally-Embedded Perspective on the Big World Hypothesis](https://mlsanity.com/paper/2512.23419/) (2025)

## Abstract (arXiv)

World models are a central component of model-based reinforcement learning. They are usually discussed in terms of what variables they predict, such as observations, rewards, states, latent or information states. We argue that there is a prior distinction: which channel they model. We consider three cases: the environment channel O: mid A:, the agent channel A: mid O:, and the realised joint process (A, O):, equivalently viewed as a channel with no inputs. Using computational mechanics, we define canonical predictive models for these three cases as ε-transducers or ε-machines. Canonical environment models recover standard predictive state representations, while the other two give analogous notions of canonical models for the agent and the joint system. We then build canonical support-restricted environment and agent models induced by closed-loop coupling, whose predictive equivalences range over continuations supported by the realised interaction. The key structural result is that canonical support-restricted environment states factor through the canonical joint causal states, and their transition structure is induced directly from the joint model; the agent-side construction is dual. Finally, we give a POMDP/controller example in which the unrestricted environment model has infinitely many states while the canonical support-restricted model induced by the coupling is finite. The framework clarifies what different world models are models of, and how coupling and support restriction can change their canonical predictive structure and complexity.