# Discovery of Useful Questions as Auxiliary Tasks

arXiv: https://arxiv.org/abs/1909.04607 · HTML: https://mlsanity.com/paper/1909.04607/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 30 of 45)

- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Reinforcement Learning, Bit by Bit](https://mlsanity.com/paper/2103.04047/) (2021)
- [Return-Based Contrastive Representation Learning for Reinforcement Learning](https://mlsanity.com/paper/2102.10960/) (2021)
- [Which Mutual-Information Representation Learning Objectives are Sufficient for Control?](https://mlsanity.com/paper/2106.07278/) (2021)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)
- [In-context Exploration-Exploitation for Reinforcement Learning](https://mlsanity.com/paper/2403.06826/) (2024)
- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning](https://mlsanity.com/paper/2112.15400/) (2021)
- [Exploring through Random Curiosity with General Value Functions](https://mlsanity.com/paper/2211.10282/) (2022)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [Continual Auxiliary Task Learning](https://mlsanity.com/paper/2202.11133/) (2022)
- [PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigation](https://mlsanity.com/paper/2305.11918/) (2023)
- [A History of Meta-gradient: Gradient Methods for Meta-learning](https://mlsanity.com/paper/2202.09701/) (2022)
- [Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning](https://mlsanity.com/paper/2206.11396/) (2022)
- [Composing Task Knowledge with Modular Successor Feature Approximators](https://mlsanity.com/paper/2301.12305/) (2023)
- [Learning State Representations from Random Deep Action-conditional Predictions](https://mlsanity.com/paper/2102.04897/) (2021)
- [One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/2111.00206/) (2021)
- [Open-Ended Reinforcement Learning with Neural Reward Functions](https://mlsanity.com/paper/2202.08266/) (2022)
- [An Empirical Study on the Power of Future Prediction in Partially Observable Environments](https://mlsanity.com/paper/2402.07102/) (2024)
- [Aligned Multi Objective Optimization](https://mlsanity.com/paper/2502.14096/) (2025)
- [Adaptive Pairwise Weights for Temporal Credit Assignment](https://mlsanity.com/paper/2102.04999/) (2021)
- [Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.12957/) (2021)
- [Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2104.04174/) (2021)
- [Neural Auto-Curricula](https://mlsanity.com/paper/2106.02745/) (2021)
- [Adapting the Function Approximation Architecture in Online Reinforcement Learning](https://mlsanity.com/paper/2106.09776/) (2021)
- [Auxiliary task discovery through generate-and-test](https://mlsanity.com/paper/2210.14361/) (2022)

## Abstract (arXiv)

Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value functions or GVFs, a fairly rich form of knowledge representation. Specifically, our method uses non-myopic meta-gradients to learn GVF-questions such that learning answers to them, as an auxiliary task, induces useful representations for the main task faced by the RL agent. We demonstrate that auxiliary tasks based on the discovered GVFs are sufficient, on their own, to build representations that support main task learning, and that they do so better than popular hand-designed auxiliary tasks from the literature. Furthermore, we show, in the context of Atari 2600 videogames, how such auxiliary tasks, meta-learned alongside the main task, can improve the data efficiency of an actor-critic agent.