# Intrinsically-Motivated Humans and Agents in Open-World Exploration

arXiv: https://arxiv.org/abs/2503.23631 · HTML: https://mlsanity.com/paper/2503.23631/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 66 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 26 of 26)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Investigating Human Priors for Playing Video Games](https://mlsanity.com/paper/1802.10217/) (2018)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Changing the Environment Based on Empowerment as Intrinsic Motivation](https://mlsanity.com/paper/1406.1767/) (2014)
- [Learning Causal Overhypotheses through Exploration in Children and Computational Models](https://mlsanity.com/paper/2202.10430/) (2022)
- [Exploring Exploration: Comparing Children with RL Agents in Unified Environments](https://mlsanity.com/paper/2005.02880/) (2020)
- [Evaluating Agents without Rewards](https://mlsanity.com/paper/2012.11538/) (2020)
- [BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping](https://mlsanity.com/paper/2409.05358/) (2024)

## Built on by (most cited first, 8 of 8)

- [From Curiosity to Competence: How World Models Interact with the Dynamics of Exploration](https://mlsanity.com/paper/2507.08210/) (2025)
- [Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models](https://mlsanity.com/paper/2505.07815/) (2025)
- [Agent-centric learning: from external reward maximization to internal knowledge curation](https://mlsanity.com/paper/2507.22255/) (2025)
- [When Assisting One Disempowers Another](https://mlsanity.com/paper/2511.04177/) (2025)
- [What Is Worth Representing? Representational Empowerment for Continual Model Construction](https://mlsanity.com/paper/2609.02322/) (2026)
- [Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction](https://mlsanity.com/paper/2510.15422/) (2025)
- [Open-Endedness Bench: Measuring Epistemic Process from Agent Records](https://mlsanity.com/paper/2610.02588/) (2026)
- [Scaling Empowered Agents](https://mlsanity.com/paper/2610.13002/) (2026)

## Abstract (arXiv)

What drives exploration? Understanding intrinsic motivation is a long-standing challenge in both cognitive science and artificial intelligence; numerous objectives have been proposed and used to train agents, yet there remains a gap between human and agent exploration. We directly compare adults, children, and AI agents in a complex open-ended environment, Crafter, and study how common intrinsic objectives: Entropy, Information Gain, and Empowerment, relate to their behavior. We find that only Entropy and Empowerment are consistently positively correlated with human exploration progress, indicating that these objectives may better inform intrinsic reward design for agents. Furthermore, across agents and humans we observe that Entropy initially increases rapidly, then plateaus, while Empowerment increases continuously, suggesting that state diversity may provide more signal in early exploration, while advanced exploration should prioritize control. Finally, we find preliminary evidence that private speech utterances, and particularly goal verbalizations, may aid exploration in children. Our data is available at https://github.com/alyd/humans_in_crafter_data.