# Meta-learning curiosity algorithms

arXiv: https://arxiv.org/abs/2003.05325 · HTML: https://mlsanity.com/paper/2003.05325/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2020.

## Built on by (most cited first, 25 of 25)

- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Bootstrapped Meta-Learning](https://mlsanity.com/paper/2109.04504/) (2021)
- [Discovering Preference Optimization Algorithms with and for Large Language Models](https://mlsanity.com/paper/2406.08414/) (2024)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Introducing Symmetries to Black Box Meta Reinforcement Learning](https://mlsanity.com/paper/2109.10781/) (2021)
- [Discovering Temporally-Aware Reinforcement Learning Algorithms](https://mlsanity.com/paper/2402.05828/) (2024)
- [Fine-grained Recognition with Learnable Semantic Data Augmentation](https://mlsanity.com/paper/2309.00399/) (2023)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design](https://mlsanity.com/paper/2310.02782/) (2023)
- [Unsupervised Meta-Learning via Few-shot Pseudo-supervised Contrastive Learning](https://mlsanity.com/paper/2303.00996/) (2023)
- [A Simple Guard for Learned Optimizers](https://mlsanity.com/paper/2201.12426/) (2022)
- [Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation](https://mlsanity.com/paper/2510.08962/) (2025)
- [BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping](https://mlsanity.com/paper/2409.05358/) (2024)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [Unified Functional Hashing in Automatic Machine Learning](https://mlsanity.com/paper/2302.05433/) (2023)
- [Agent-centric learning: from external reward maximization to internal knowledge curation](https://mlsanity.com/paper/2507.22255/) (2025)
- [Learning Interestingness in Automated Mathematical Theory Formation](https://mlsanity.com/paper/2511.14778/) (2025)
- [A Brief Summary of Interactions Between Meta-Learning and Self-Supervised Learning](https://mlsanity.com/paper/2103.00845/) (2021)
- [Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research](https://mlsanity.com/paper/2107.03015/) (2021)
- [Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stability](https://mlsanity.com/paper/2204.04292/) (2022)
- [An Adaptive Approach for Probabilistic Wind Power Forecasting Based on Meta-Learning](https://mlsanity.com/paper/2308.07980/) (2023)
- [Black Box Meta-Learning Intrinsic Rewards](https://mlsanity.com/paper/2407.21546/) (2024)
- [Curiosity creates Diversity in Policy Search](https://mlsanity.com/paper/2212.03530/) (2022)

## Abstract (arXiv)

We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs, acrobot, lunar lander, ant and hopper.