# Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

arXiv: https://arxiv.org/abs/2107.12544 · HTML: https://mlsanity.com/paper/2107.12544/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 89 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 27 of 27)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://mlsanity.com/paper/1507.08750/) (2015)
- [Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models](https://mlsanity.com/paper/1507.00814/) (2015)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [Building machines that adapt and compute like brains](https://mlsanity.com/paper/1711.04203/) (2017)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [Recurrent Environment Simulators](https://mlsanity.com/paper/1704.02254/) (2017)
- [Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning](https://mlsanity.com/paper/1907.09620/) (2019)
- [Learning Visual Predictive Models of Physics for Playing Billiards](https://mlsanity.com/paper/1511.07404/) (2015)
- [Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics](https://mlsanity.com/paper/1706.04317/) (2017)
- [COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration](https://mlsanity.com/paper/1905.09275/) (2019)
- [Causal Reasoning from Meta-reinforcement Learning](https://mlsanity.com/paper/1901.08162/) (2019)
- [Towards Deep Symbolic Reinforcement Learning](https://mlsanity.com/paper/1609.05518/) (2016)
- [Learning to Play with Intrinsically-Motivated Self-Aware Agents](https://mlsanity.com/paper/1802.07442/) (2018)
- [Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening](https://mlsanity.com/paper/1611.01606/) (2016)
- [Learning sparse relational transition models](https://mlsanity.com/paper/1810.11177/) (2018)
- [A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games](https://mlsanity.com/paper/1611.07078/) (2016)

## Built on by (most cited first, 30 of 43)

- [Language Models as Agent Models](https://mlsanity.com/paper/2212.01681/) (2022)
- [WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment](https://mlsanity.com/paper/2402.12275/) (2024)
- [Policy Expansion for Bridging Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2302.00935/) (2023)
- [MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks](https://mlsanity.com/paper/2310.19677/) (2023)
- [Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search](https://mlsanity.com/paper/2405.15383/) (2024)
- [Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation](https://mlsanity.com/paper/2206.11403/) (2022)
- [PoE-World: Compositional World Modeling with Products of Programmatic Experts](https://mlsanity.com/paper/2505.10819/) (2025)
- [Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning](https://mlsanity.com/paper/2402.06025/) (2024)
- [Human-like Few-Shot Learning via Bayesian Reasoning over Natural Language](https://mlsanity.com/paper/2306.02797/) (2023)
- [ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles](https://mlsanity.com/paper/2306.06871/) (2023)
- [Assessing Adaptive World Models in Machines with Novel Games](https://mlsanity.com/paper/2507.12821/) (2025)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Synthesizing world models for bilevel planning](https://mlsanity.com/paper/2503.20124/) (2025)
- [Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models](https://mlsanity.com/paper/2507.12547/) (2025)
- [WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-making](https://mlsanity.com/paper/2506.06725/) (2025)
- [AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games](https://mlsanity.com/paper/2602.17594/) (2026)
- [Modeling Others' Minds as Code](https://mlsanity.com/paper/2510.01272/) (2025)
- [Learning to solve complex tasks by growing knowledge culturally across generations](https://mlsanity.com/paper/2107.13377/) (2021)
- [Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning](https://mlsanity.com/paper/2204.01437/) (2022)
- [Does Deep Learning Learn to Abstract? A Systematic Probing Framework](https://mlsanity.com/paper/2302.11978/) (2023)
- [Target Languages (vs. Inductive Biases) for Learning to Act and Plan](https://mlsanity.com/paper/2109.07195/) (2021)
- [Regularity as Intrinsic Reward for Free Play](https://mlsanity.com/paper/2312.01473/) (2023)
- [Credit Assignment: Challenges and Opportunities in Developing Human-like AI Agents](https://mlsanity.com/paper/2307.08171/) (2023)
- [Predictive representations: building blocks of intelligence](https://mlsanity.com/paper/2402.06590/) (2024)
- [The Discovery Engine: A Framework for AI-Driven Synthesis and Navigation of Scientific Knowledge Landscapes](https://mlsanity.com/paper/2505.17500/) (2025)
- [Possible Principles for Aligned Structure Learning Agents](https://mlsanity.com/paper/2410.00258/) (2024)
- [Active inference and artificial reasoning](https://mlsanity.com/paper/2512.21129/) (2025)
- [Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents](https://mlsanity.com/paper/2602.00929/) (2026)
- [Generalised agent for solving higher board states of tic tac toe using Reinforcement Learning](https://mlsanity.com/paper/2212.12252/) (2022)
- [Learning to Play Video Games with Intuitive Physics Priors](https://mlsanity.com/paper/2409.13886/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) studies how an agent comes to achieve reward in an environment through interactions over time. Recent advances in machine RL have surpassed human expertise at the world's oldest board games and many classic video games, but they require vast quantities of experience to learn successfully -- none of today's algorithms account for the human ability to learn so many different tasks, so quickly. Here we propose a new approach to this challenge based on a particularly strong form of model-based RL which we call Theory-Based Reinforcement Learning, because it uses human-like intuitive theories -- rich, abstract, causal models of physical objects, intentional agents, and their interactions -- to explore and model an environment, and plan effectively to achieve task goals. We instantiate the approach in a video game playing agent called EMPA (the Exploring, Modeling, and Planning Agent), which performs Bayesian inference to learn probabilistic generative models expressed as programs for a game-engine simulator, and runs internal simulations over these models to support efficient object-based, relational exploration and heuristic planning. EMPA closely matches human learning efficiency on a suite of 90 challenging Atari-style video games, learning new games in just minutes of game play and generalizing robustly to new game situations and new levels. The model also captures fine-grained structure in people's exploration trajectories and learning dynamics. Its design and behavior suggest a way forward for building more general human-like AI systems.