# What Can Learned Intrinsic Rewards Capture?

arXiv: https://arxiv.org/abs/1912.05500 · HTML: https://mlsanity.com/paper/1912.05500/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2019.

## Built on by (most cited first, 30 of 36)

- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Adversarial Intrinsic Motivation for Reinforcement Learning](https://mlsanity.com/paper/2105.13345/) (2021)
- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [Redeeming Intrinsic Rewards via Constrained Optimization](https://mlsanity.com/paper/2211.07627/) (2022)
- [A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health](https://mlsanity.com/paper/2402.14807/) (2024)
- [Hypernetworks for Zero-shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/2211.15457/) (2022)
- [Behavior Alignment via Reward Function Optimization](https://mlsanity.com/paper/2310.19007/) (2023)
- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)
- [A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning](https://mlsanity.com/paper/2112.15400/) (2021)
- [Learning Sketches for Decomposing Planning Problems into Subproblems of Bounded Width: Extended Version](https://mlsanity.com/paper/2203.14852/) (2022)
- [LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward](https://mlsanity.com/paper/2210.05409/) (2022)
- [CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning](https://mlsanity.com/paper/2306.13412/) (2023)
- [Expressing and Exploiting the Common Subgoal Structure of Classical Planning Domains Using Sketches: Extended Version](https://mlsanity.com/paper/2105.04250/) (2021)
- [Wasserstein Distance Maximizing Intrinsic Control](https://mlsanity.com/paper/2110.15331/) (2021)
- [Learning Synthetic Environments and Reward Networks for Reinforcement Learning](https://mlsanity.com/paper/2202.02790/) (2022)
- [General Policies, Subgoal Structure, and Planning Width](https://mlsanity.com/paper/2311.05490/) (2023)
- [Target Languages (vs. Inductive Biases) for Learning to Act and Plan](https://mlsanity.com/paper/2109.07195/) (2021)
- [An Investigation of the Bias-Variance Tradeoff in Meta-Gradients](https://mlsanity.com/paper/2209.11303/) (2022)
- [Hindsight Foresight Relabeling for Meta-Reinforcement Learning](https://mlsanity.com/paper/2109.09031/) (2021)
- [Adaptive Pairwise Weights for Temporal Credit Assignment](https://mlsanity.com/paper/2102.04999/) (2021)
- [Neural Auto-Curricula](https://mlsanity.com/paper/2106.02745/) (2021)
- [Evolution of Rewards for Food and Motor Action by Simulating Birth and Death](https://mlsanity.com/paper/2406.15016/) (2024)
- [Sketch Decompositions for Classical Planning via Deep Reinforcement Learning](https://mlsanity.com/paper/2412.08574/) (2024)
- [Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization](https://mlsanity.com/paper/2605.06145/) (2026)
- [Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs](https://mlsanity.com/paper/2306.15903/) (2023)
- [Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback](https://mlsanity.com/paper/2403.11330/) (2024)
- [Black Box Meta-Learning Intrinsic Rewards](https://mlsanity.com/paper/2407.21546/) (2024)
- [Delayed Rewards Calibration via Reward Empirical Sufficiency](https://mlsanity.com/paper/2102.10527/) (2021)

## Abstract (arXiv)

The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful intrinsic reward functions across multiple lifetimes of experience. Through several proof-of-concept experiments, we show that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. Furthermore, we show that unlike policy transfer methods that capture "how" the agent should behave, the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment by capturing "what" the agent should strive to do.