mlsanityData to 14 Oct 2026

Discovery of Useful Questions as Auxiliary Tasks

arXiv 1909.04607 ↗First posted 10 Sep 2019cs.AI cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value functions or GVFs, a fairly rich form of knowledge representation. Specifically, our method uses non-myopic meta-gradients to learn GVF-questions such that learning answers to them, as an auxiliary task, induces useful representations for the main task faced by the RL agent. We demonstrate that auxiliary tasks based on the discovered GVFs are sufficient, on their own, to build representations that support main task learning, and that they do so better than popular hand-designed auxiliary tasks from the literature. Furthermore, we show, in the context of Atari 2600 videogames, how such auxiliary tasks, meta-learned alongside the main task, can improve the data efficiency of an actor-critic agent.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 14; 2023: 6; 2024: 2; 2025: 4; 2026: 2.

45 papers.

Sort:

By area:

  1. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  2. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  3. 2021Reinforcement Learning, Bit by Bitcited by 59
  4. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  5. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 29
  6. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  7. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 17
  8. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
  9. 2025Interpretable Machine Learning in Physics: A Reviewcited by 14
  10. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  11. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 13
  12. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 12
33 more
  1. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  2. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  3. 2022Continual Auxiliary Task Learningcited by 9
  4. 2023PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigationcited by 8
  5. 2022A History of Meta-gradient: Gradient Methods for Meta-learningcited by 7
  6. 2022Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learningcited by 7
  7. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7
  8. 2021Learning State Representations from Random Deep Action-conditional Predictionscited by 6
  9. 2021One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learningcited by 5
  10. 2022Open-Ended Reinforcement Learning with Neural Reward Functionscited by 5
  11. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 4
  12. 2025Aligned Multi Objective Optimizationcited by 3
  13. 2021Adaptive Pairwise Weights for Temporal Credit Assignmentcited by 2
  14. 2021Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learningcited by 2
  15. 2021Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learningcited by 2
  16. 2021Neural Auto-Curriculacited by 2
  17. 2021Adapting the Function Approximation Architecture in Online Reinforcement Learningcited by 2
  18. 2022Auxiliary task discovery through generate-and-testcited by 2
  19. 2023Scaling Goal-based Exploration via Pruning Proto-goalscited by 2
  20. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2
  21. 2021Parameter-free Gradient Temporal Difference Learningcited by 1
  22. 2021Finding Useful Predictions by Meta-gradient Descent to Improve Decision-makingcited by 1
  23. 2022Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learningcited by 1
  24. 2022What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experiencecited by 1
  25. 2023Discovering Object-Centric Generalized Value Functions From Pixelscited by 1
  26. 2025Simple Optimizers for Convex Aligned Multi-Objective Optimizationcited by 1
  27. 2021Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloningcited by 0
  28. 2022elBERto: Self-supervised Commonsense Learning for Question Answeringcited by 0
  29. 2022Learning to Accelerate by the Methods of Step-size Planningcited by 0
  30. 2022Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Functioncited by 0
  31. 2023Acceleration in Policy Optimizationcited by 0
  32. 2025Generating Auxiliary Tasks with Reinforcement Learningcited by 0
  33. 2026Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Tradingcited by 0