mlsanityData to 7 Oct 2026

On The Effect of Auxiliary Tasks on Representation Dynamics

arXiv 2102.13089 ↗First posted 25 Feb 2021cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 37 of its 72 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

37 of 72 references.

References · 37

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  12. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
25 more
  1. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  2. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  3. 2016Learning to Navigate in Complex Environmentscited by 192
  4. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  5. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  6. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  7. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  8. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 124
  9. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  10. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  11. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  12. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 98
  13. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  14. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  15. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  16. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
  17. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  18. 2019SciPy 1.0--Fundamental Algorithms for Scientific Computing in Pythoncited by 49
  19. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  20. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  21. 2019Comments on the Du-Kakade-Wang-Yang Lower Boundscited by 16
  22. 2017Shallow Updates for Deep Reinforcement Learningcited by 14
  23. 2018Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementationscited by 12
  24. 2019Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learningcited by 11
  25. 2017Unsupervised Basis Function Adaptation for Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 16; 2023: 14; 2024: 13; 2025: 5; 2026: 3.

56 papers.

Sort:

By area:

  1. 2022Multi-Game Decision Transformerscited by 212
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  3. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  4. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  5. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  6. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  7. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 40
  8. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  9. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  10. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  11. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
  12. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
38 more
  1. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  2. 2022Transfer RL across Observation Feature Spaces via Model-Based Regularizationcited by 18
  3. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  4. 2023TD Convergence: An Optimization Perspectivecited by 15
  5. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 13
  6. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  7. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  8. 2024Investigating the Histogram Loss in Regressioncited by 11
  9. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  10. 2022Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learningcited by 8
  11. 2023Policy Gradient Methods in the Presence of Symmetries and State Abstractionscited by 8
  12. 2022A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspacescited by 7
  13. 2023Predictive auxiliary objectives in deep RL mimic learning in the braincited by 7
  14. 2024Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equationcited by 7
  15. 2024A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learningcited by 7
  16. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  17. 2024Random Latent Exploration for Deep Reinforcement Learningcited by 6
  18. 2021An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybulletcited by 5
  19. 2022What makes useful auxiliary tasks in reinforcement learning: investigating the effect of the target policycited by 5
  20. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  21. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
  22. 2023PcLast: Discovering Plannable Continuous Latent Statescited by 4
  23. 2022Engineering flexible machine learning systems by traversing functionally-invariant pathscited by 3
  24. 2023Towards a Better Understanding of Representation Dynamics under TD-learningcited by 3
  25. 2023Investigating Navigation Strategies in the Morris Water Maze through Deep Reinforcement Learningcited by 3
  26. 2022Auxiliary task discovery through generate-and-testcited by 2
  27. 2023Representations and Exploration for Deep Reinforcement Learning using Singular Value Decompositioncited by 2
  28. 2023Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learningcited by 2
  29. 2024A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agentscited by 2
  30. 2021Balancing Value Underestimation and Overestimation with Realistic Actor-Criticcited by 1
  31. 2022VI-PINNs: Variance-involved Physics-informed Neural Networks for Fast and Accurate Prediction of Partial Differential Equationscited by 1
  32. 2023The Ladder in Chaos: A Simple and Effective Improvement to General DRL Algorithms by Policy Path Trimming and Boostingcited by 1
  33. 2024Learning the Target Network in Function Spacecited by 1
  34. 2025Generalized Task-Driven Medical Image Quality Enhancement with Gradient Promotioncited by 1
  35. 2025Deep Double Q-learningcited by 1
  36. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  37. 2021Uncertainty-aware Low-Rank Q-Matrix Estimation for Deep Reinforcement Learningcited by 0
  38. 2022The Sandbox Environment for Generalizable Agent Research (SEGAR)cited by 0

50 most cited of 56 All 56 (JSON)