# Modeling Human Exploration Through Resource-Rational Reinforcement Learning

arXiv: https://arxiv.org/abs/2201.11817 · HTML: https://mlsanity.com/paper/2201.11817/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [Learning to Optimize via Information-Directed Sampling](https://mlsanity.com/paper/1403.5556/) (2014)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Meta-Learning without Memorization](https://mlsanity.com/paper/1912.03820/) (2019)
- [PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees](https://mlsanity.com/paper/2002.05551/) (2020)
- [Meta-learning of Sequential Strategies](https://mlsanity.com/paper/1905.03030/) (2019)
- [Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters](https://mlsanity.com/paper/1810.00440/) (2018)
- [Robust Predictable Control](https://mlsanity.com/paper/2109.03214/) (2021)
- [Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization](https://mlsanity.com/paper/2011.02872/) (2020)

## Built on by (most cited first, 13 of 13)

- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [Meta-in-context learning in large language models](https://mlsanity.com/paper/2305.12907/) (2023)
- [In-context learning agents are asymmetric belief updaters](https://mlsanity.com/paper/2402.03969/) (2024)
- [Reinforcement Learning with Simple Sequence Priors](https://mlsanity.com/paper/2305.17109/) (2023)
- [Predictive auxiliary objectives in deep RL mimic learning in the brain](https://mlsanity.com/paper/2310.06089/) (2023)
- [Human-like Category Learning by Injecting Ecological Priors from Large Language Models into Neural Networks](https://mlsanity.com/paper/2402.01821/) (2024)
- [Bayesian Reinforcement Learning with Limited Cognitive Load](https://mlsanity.com/paper/2305.03263/) (2023)
- [Meta-learning ecological priors from large language models explains human learning and decision making](https://mlsanity.com/paper/2509.00116/) (2025)
- [Program-Based Strategy Induction for Reinforcement Learning](https://mlsanity.com/paper/2402.16668/) (2024)
- [Simulating Human Audiovisual Search Behavior](https://mlsanity.com/paper/2602.02790/) (2026)
- [CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention](https://mlsanity.com/paper/2301.06216/) (2023)
- [Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light](https://mlsanity.com/paper/2507.11482/) (2025)
- [Mental Accounts for Actions: EWA-Inspired Attention in Decision Transformers](https://mlsanity.com/paper/2509.15498/) (2025)

## Abstract (arXiv)

Equipping artificial agents with useful exploration mechanisms remains a challenge to this day. Humans, on the other hand, seem to manage the trade-off between exploration and exploitation effortlessly. In the present article, we put forward the hypothesis that they accomplish this by making optimal use of limited computational resources. We study this hypothesis by meta-learning reinforcement learning algorithms that sacrifice performance for a shorter description length (defined as the number of bits required to implement the given algorithm). The emerging class of models captures human exploration behavior better than previously considered approaches, such as Boltzmann exploration, upper confidence bound algorithms, and Thompson sampling. We additionally demonstrate that changing the description length in our class of models produces the intended effects: reducing description length captures the behavior of brain-lesioned patients while increasing it mirrors cognitive development during adolescence.