mlsanityData to 14 Oct 2026

Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks

arXiv 2304.12567 ↗First posted 25 Apr 2023cs.LG cs.AI stat.MLcited by 42

Not on the map. Why?

Cited by 29 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Auxiliary tasks improve the representations learned by deep reinforcement learning agents. Analytically, their effect is reasonably well understood; in practice, however, their primary use remains in support of a main learning objective, rather than as a method for learning representations. This is perhaps surprising given that many auxiliary tasks are defined procedurally, and hence can be treated as an essentially infinite source of information about the environment. Based on this observation, we study the effectiveness of auxiliary tasks for learning rich representations, focusing on the setting where the number of tasks and the size of the agent's network are simultaneously increased. For this purpose, we derive a new family of auxiliary tasks based on the successor measure. These tasks are easy to implement and have appealing theoretical properties. Combined with a suitable off-policy learning rule, the result is a representation learning algorithm that can be understood as extending Mahadevan & Maggioni (2007)'s proto-value functions to deep reinforcement learning -- accordingly, we call the resulting object proto-value networks. Through a series of experiments on the Arcade Learning Environment, we demonstrate that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms, using only linear approximation and a small number (~4M) of interactions with the environment's reward function.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

37 of 76 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2019Model-Based Reinforcement Learning for Ataricited by 502
  9. 2020Reinforcement Learning with Augmented Datacited by 468
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
25 more
  1. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  2. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  4. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  5. 2019Unsupervised State Representation Learning in Ataricited by 144
  6. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  7. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  8. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  9. 2021Learning One Representation to Optimize All Rewardscited by 115
  10. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  11. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  12. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  13. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  14. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  15. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 60
  16. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  17. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
  18. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  19. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  20. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  21. 2018Spectral Inference Networks: Unifying Deep and Spectral Learningcited by 35
  22. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  23. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  24. 2022Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RLcited by 7
  25. 2022A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspacescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 16; 2025: 11; 2026: 8.

42 papers.

Sort:

By area:

  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  3. 2024Foundation Policies with Hilbert Representationscited by 94
  4. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  5. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  6. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  7. 2023Small batch deep reinforcement learningcited by 36
  8. 2025Temporal Difference Flowscited by 24
  9. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  10. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  11. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  12. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
30 more
  1. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 20
  2. 2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
  3. 2024A Distributional Analogue to the Successor Representationcited by 17
  4. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  5. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  6. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 14
  7. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  8. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
  9. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  10. 2023Proper Laplacian Representation Learningcited by 11
  11. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
  12. 2024Learning Successor Features the Simple Waycited by 10
  13. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  14. 2025Multi-Task Reinforcement Learning Enables Parameter Scalingcited by 8
  15. 2024CALE: Continuous Arcade Learning Environmentcited by 5
  16. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  17. 2024Predictive representations: building blocks of intelligencecited by 4
  18. 2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 4
  19. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  20. 2025Spectral Bellman Method: Unifying Representation and Exploration in RLcited by 4
  21. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  22. 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewardscited by 2
  23. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2
  24. 2026The Laplacian Keyboard: Beyond the Linear Spancited by 1
  25. 2026Impact of Connectivity on Laplacian Representations in Reinforcement Learningcited by 1
  26. 2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 1
  27. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  28. 2025Online Training and Pruning of Deep Reinforcement Learning Networkscited by 0
  29. 2026DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learningcited by 0
  30. 2026Mastering Atari 2600 Games with Discovered Optionscited by 0