mlsanityData to 7 Oct 2026

When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning

arXiv 2406.17718 ↗First posted 25 Jun 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We investigate the impact of auxiliary learning tasks such as observation reconstruction and latent self-prediction on the representation learning problem in reinforcement learning. We also study how they interact with distractions and observation functions in the MDP. We provide a theoretical analysis of the learning dynamics of observation reconstruction, latent self-prediction, and TD learning in the presence of distractions and observation functions under linear model assumptions. With this formalization, we are able to explain why latent-self prediction is a helpful auxiliary task, while observation reconstruction can provide more useful features when used in isolation. Our empirical analysis shows that the insights obtained from our learning dynamics framework predicts the behavior of these loss functions beyond the linear model assumption in non-linear neural networks. This reinforces the usefulness of the linear model framework not only for theoretical analysis, but also practical benefit for applied problems.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

33 of 45 references.

References · 33

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  12. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
21 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  3. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  4. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  5. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  6. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  7. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  8. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  9. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  10. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  11. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 40
  12. 2021Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiationcited by 35
  13. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  14. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  15. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  16. 2020Regularized linear autoencoders recover the principal components, eventuallycited by 26
  17. 2021Metrics and continuity in reinforcement learningcited by 26
  18. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  19. 2018Learning Dynamics of Linear Denoising Autoencoderscited by 16
  20. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  21. 2023Towards a Better Understanding of Representation Dynamics under TD-learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  2. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 17
  3. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  4. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  5. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  6. 2026The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Workscited by 3
  7. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Modelscited by 2
  8. 2026Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Predictioncited by 2
  9. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  10. 2026NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learningcited by 0
  11. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  12. 2026What a World Model Represents Is Three Questionscited by 0
1 more
  1. 2026FERPO: Forward Entropy-Regularized Policy Optimizationcited by 0