# A Theory of Abstraction in Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.00397 · HTML: https://mlsanity.com/paper/2203.00397/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2022.

## Built on by (most cited first, 27 of 27)

- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation](https://mlsanity.com/paper/2210.12631/) (2022)
- [Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learning](https://mlsanity.com/paper/2206.02072/) (2022)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [Towards a Grounded Theory of Causation for Embodied AI](https://mlsanity.com/paper/2206.13973/) (2022)
- [Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction](https://mlsanity.com/paper/2212.07486/) (2022)
- [Planning to the Information Horizon of BAMDPs via Epistemic State Abstraction](https://mlsanity.com/paper/2210.16872/) (2022)
- [Off-policy Evaluation with Deeply-abstracted States](https://mlsanity.com/paper/2406.19531/) (2024)
- [Satisficing Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/2407.12185/) (2024)
- [Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments](https://mlsanity.com/paper/2512.19154/) (2025)
- [The Myhill-Nerode Theorem for Bounded Interaction: Canonical Abstractions via Agent-Bounded Indistinguishability](https://mlsanity.com/paper/2603.21399/) (2026)
- [Abstract-to-Executable Trajectory Translation for One-Shot Task Generalization](https://mlsanity.com/paper/2210.07658/) (2022)
- [Scaling Goal-based Exploration via Pruning Proto-goals](https://mlsanity.com/paper/2302.04693/) (2023)
- [Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction](https://mlsanity.com/paper/2408.11816/) (2024)
- [Characterizing the Action-Generalization Gap in Deep Q-Learning](https://mlsanity.com/paper/2205.05588/) (2022)
- [Causal prompting model-based offline reinforcement learning](https://mlsanity.com/paper/2406.01065/) (2024)
- [Multi-Agent Reinforcement Learning in Cybersecurity: From Fundamentals to Applications](https://mlsanity.com/paper/2505.19837/) (2025)
- [Analogy making as amortised model construction](https://mlsanity.com/paper/2507.16511/) (2025)
- [Multi-layer Abstraction for Nested Generation of Options (MANGO) in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2508.17751/) (2025)
- [Scaling Intelligent Agents in Combat Simulations for Wargaming](https://mlsanity.com/paper/2402.06694/) (2024)
- [Spatio-temporal Value Semantics-based Abstraction for Dense Deep Reinforcement Learning](https://mlsanity.com/paper/2405.15829/) (2024)
- [Towards Faster Matrix Diagonalization with Graph Isomorphism Networks and the AlphaZero Framework](https://mlsanity.com/paper/2407.00779/) (2024)
- [Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2512.04958/) (2025)
- [Lessons from Neuroscience for AI: How integrating Actions, Compositional Structure and Episodic Memory could enable Safe, Interpretable and Human-Like AI](https://mlsanity.com/paper/2512.22568/) (2025)
- [Abstract Sim2Real through Approximate Information States](https://mlsanity.com/paper/2604.15289/) (2026)
- [Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2606.25357/) (2026)
- [World models of environment, agent and joint agent-environment systems](https://mlsanity.com/paper/2608.20401/) (2026)

## Abstract (arXiv)

Reinforcement learning defines the problem facing agents that learn to make good decisions through action and observation alone. To be effective problem solvers, such agents must efficiently explore vast worlds, assign credit from delayed feedback, and generalize to new experiences, all while making use of limited data, computational resources, and perceptual bandwidth. Abstraction is essential to all of these endeavors. Through abstraction, agents can form concise models of their environment that support the many practices required of a rational, adaptive decision maker. In this dissertation, I present a theory of abstraction in reinforcement learning. I first offer three desiderata for functions that carry out the process of abstraction: they should 1) preserve representation of near-optimal behavior, 2) be learned and constructed efficiently, and 3) lower planning or learning time. I then present a suite of new algorithms and analysis that clarify how agents can learn to abstract according to these desiderata. Collectively, these results provide a partial path toward the discovery and use of abstraction that minimizes the complexity of effective reinforcement learning.