# AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretability

arXiv: https://arxiv.org/abs/2504.04608 · HTML: https://mlsanity.com/paper/2504.04608/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 32 of its 98 references that are in the map.
- First posted on arXiv 6 Apr 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 30 of 32)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Approximate information state for approximate planning and reinforcement learning in partially observed systems](https://mlsanity.com/paper/2010.08843/) (2020)
- [Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation](https://mlsanity.com/paper/2305.02231/) (2023)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy](https://mlsanity.com/paper/2402.04247/) (2024)
- [Learning Causal State Representations of Partially Observable Environments](https://mlsanity.com/paper/1906.10437/) (2019)
- [Security of AI Agents](https://mlsanity.com/paper/2406.08689/) (2024)
- [International Scientific Report on the Safety of Advanced AI (Interim Report)](https://mlsanity.com/paper/2412.05282/) (2024)
- [A Complexity-Based Theory of Compositionality](https://mlsanity.com/paper/2410.14817/) (2024)
- [Generative models as parsimonious descriptions of sensorimotor loops](https://mlsanity.com/paper/1904.12937/) (2019)
- [Shannon Entropy Rate of Hidden Markov Processes](https://mlsanity.com/paper/2008.12886/) (2020)
- [Minimal Realization Problems for Hidden Markov Models](https://mlsanity.com/paper/1411.3698/) (2014)
- [Interpreting systems as solving POMDPs: a step towards a formal understanding of agency](https://mlsanity.com/paper/2209.01619/) (2022)
- [An active inference implementation of phototaxis](https://mlsanity.com/paper/1707.01806/) (2017)
- [Topology, Convergence, and Reconstruction of Predictive States](https://mlsanity.com/paper/2109.09203/) (2021)
- [Interpreting Dynamical Systems as Bayesian Reasoners](https://mlsanity.com/paper/2112.13523/) (2021)
- [A Bayesian Interpretation of the Internal Model Principle](https://mlsanity.com/paper/2503.00511/) (2025)
- [Evaluating Compositional Scene Understanding in Multimodal Generative Models](https://mlsanity.com/paper/2503.23125/) (2025)
- [On the Realization of Hidden Markov Models and Tensor Decomposition](https://mlsanity.com/paper/2008.11487/) (2020)

## Built on by (most cited first, 3 of 3)

- [Identifiability and minimality bounds of quantum and post-quantum models of classical stochastic processes](https://mlsanity.com/paper/2509.03004/) (2025)
- [Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2606.25357/) (2026)
- [Approximate Homomorphisms and Convergent Representations in Transducers](https://mlsanity.com/paper/2608.20428/) (2026)

## Abstract (arXiv)

Recent work proposes using world models to generate controlled virtual environments in which AI agents can be tested before deployment to ensure their reliability and safety. However, accurate world models often have high computational demands that can severely restrict the scope and depth of such assessments. Inspired by the classic `brain in a vat' thought experiment, here we investigate ways of simplifying world models that remain agnostic to the AI agent under evaluation. By following principles from computational mechanics, our approach reveals a fundamental trade-off in world model construction between efficiency and interpretability, demonstrating that no single world model can optimise all desirable characteristics. Building on this trade-off, we identify procedures to build world models that either minimise memory requirements, delineate the boundaries of what is learnable, or allow tracking causes of undesirable outcomes. In doing so, this work establishes fundamental limits in world modelling, leading to actionable guidelines that inform core design choices related to effective agent evaluation.