mlsanityData to 7 Oct 2026

Continual Auxiliary Task Learning

arXiv 2202.11133 ↗First posted 22 Feb 2022cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off-policy learning algorithms have been developed to learn such predictions, but as yet there is little work on how to adapt the behavior to gather useful data for those off-policy predictions. In this work, we investigate a reinforcement learning system designed to learn a collection of auxiliary tasks, with a behavior policy learning to take actions to improve those auxiliary predictions. We highlight the inherent non-stationarity in this continual auxiliary task learning problem, for both prediction learners and the behavior learner. We develop an algorithm based on successor features that facilitates tracking under non-stationary rewards, and prove the separation into learning successor features and rewards provides convergence rate improvements. We conduct an in-depth study into the resulting multi-prediction learning system.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 48 of its 75 references that are in the map. First posted on arXiv 22 Feb 2022.

Builds on

48 of 75 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  11. 2016Variational Intrinsic Controlcited by 262
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
36 more
  1. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  2. 2016Learning to Navigate in Complex Environmentscited by 192
  3. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  4. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  5. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  6. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  7. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  8. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  9. 2018Universal Successor Features Approximatorscited by 102
  10. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  11. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  12. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  13. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 75
  14. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  15. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  16. 2017Rotting Banditscited by 57
  17. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 55
  18. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  19. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  20. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  21. 2016Unifying task specification in reinforcement learningcited by 41
  22. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  23. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  24. 2020Active World Model Learning with Progress Curiositycited by 27
  25. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  26. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  27. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  28. 2013Improved and Generalized Upper Bounds on the Complexity of Policy Iterationcited by 21
  29. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  30. 2017Multi-step Off-policy Learning Without Importance Sampling Ratioscited by 19
  31. 2018General Value Function Networkscited by 16
  32. 2019Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learningcited by 11
  33. 2019Meta-descent for Online, Continual Predictioncited by 10
  34. 2018Online Off-policy Predictioncited by 9
  35. 2020Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning with Polynomial Sample Complexitycited by 7
  36. 2018High-confidence error estimates for learned value functionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 5; 2025: 1; 2026: 1.

9 papers.

  1. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  2. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  3. 2025Adaptive Exploration for Multi-Reward Multi-Policy Evaluationcited by 5
  4. 2024Predictive representations: building blocks of intelligencecited by 4
  5. 2024A Benchmark Grocery Dataset of Realworld Point Clouds From Single Viewcited by 2
  6. 2024Adaptive Exploration for Data-Efficient General Value Function Evaluationscited by 2
  7. 2024Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processescited by 2
  8. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  9. 2023Discerning Temporal Difference Learningcited by 0