# Finding Useful Predictions by Meta-gradient Descent to Improve Decision-making

arXiv: https://arxiv.org/abs/2111.11212 · HTML: https://mlsanity.com/paper/2111.11212/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 17 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 3 of 3)

- [Discovery of Useful Questions as Auxiliary Tasks](https://mlsanity.com/paper/1909.04607/) (2019)
- [General Value Function Networks](https://mlsanity.com/paper/1807.06763/) (2018)
- [What's a Good Prediction? Challenges in evaluating an agent's knowledge](https://mlsanity.com/paper/2001.08823/) (2020)

## Built on by (most cited first, 1 of 1)

- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)

## Abstract (arXiv)

In computational reinforcement learning, a growing body of work seeks to express an agent's model of the world through predictions about future sensations. In this manuscript we focus on predictions expressed as General Value Functions: temporally extended estimates of the accumulation of a future signal. One challenge is determining from the infinitely many predictions that the agent could possibly make which might support decision-making. In this work, we contribute a meta-gradient descent method by which an agent can directly specify what predictions it learns, independent of designer instruction. To that end, we introduce a partially observable domain suited to this investigation. We then demonstrate that through interaction with the environment an agent can independently select predictions that resolve the partial-observability, resulting in performance similar to expertly chosen value functions. By learning, rather than manually specifying these predictions, we enable the agent to identify useful predictions in a self-supervised manner, taking a step towards truly autonomous systems.