mlsanityData to 7 Oct 2026

Modeling Human Exploration Through Resource-Rational Reinforcement Learning

arXiv 2201.11817 ↗First posted 27 Jan 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Equipping artificial agents with useful exploration mechanisms remains a challenge to this day. Humans, on the other hand, seem to manage the trade-off between exploration and exploitation effortlessly. In the present article, we put forward the hypothesis that they accomplish this by making optimal use of limited computational resources. We study this hypothesis by meta-learning reinforcement learning algorithms that sacrifice performance for a shorter description length (defined as the number of bits required to implement the given algorithm). The emerging class of models captures human exploration behavior better than previously considered approaches, such as Boltzmann exploration, upper confidence bound algorithms, and Thompson sampling. We additionally demonstrate that changing the description length in our class of models produces the intended effects: reducing description length captures the behavior of brain-lesioned patients while increasing it mirrors cognitive development during adolescence.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 24 of its 67 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

24 of 67 references.

References · 24

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  8. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2017A Tutorial on Thompson Samplingcited by 420
  11. 2016Learning to reinforcement learncited by 369
  12. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
12 more
  1. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  2. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  3. 2018Long short-term memory and learning-to-learn in networks of spiking neuronscited by 179
  4. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  5. 2021Collaborating with Humans without Human Datacited by 139
  6. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  7. 2019Meta-Learning without Memorizationcited by 92
  8. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  9. 2019Meta-learning of Sequential Strategiescited by 68
  10. 2018Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameterscited by 56
  11. 2021Robust Predictable Controlcited by 29
  12. 2020Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimizationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 3; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  2. 2023Meta-in-context learning in large language modelscited by 59
  3. 2024In-context learning agents are asymmetric belief updaterscited by 13
  4. 2023Reinforcement Learning with Simple Sequence Priorscited by 11
  5. 2023Predictive auxiliary objectives in deep RL mimic learning in the braincited by 7
  6. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  7. 2024Human-like Category Learning by Injecting Ecological Priors from Large Language Models into Neural Networkscited by 3
  8. 2025Meta-learning ecological priors from large language models explains human learning and decision makingcited by 2
  9. 2024Program-Based Strategy Induction for Reinforcement Learningcited by 1
  10. 2026Simulating Human Audiovisual Search Behaviorcited by 1
  11. 2023CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Interventioncited by 0
  12. 2025Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Lightcited by 0
1 more
  1. 2025Mental Accounts for Actions: EWA-Inspired Attention in Decision Transformerscited by 0