mlsanityData to 14 Oct 2026

Rethinking Value Function Learning for Generalization in Reinforcement Learning

arXiv 2210.09960 ↗First posted 18 Oct 2022cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Our work focuses on training RL agents on multiple visually diverse environments to improve observational generalization performance. In prior methods, policy and value networks are separately optimized using a disjoint network architecture to avoid interference and obtain a more accurate value function. We identify that a value network in the multi-environment setting is more challenging to optimize and prone to memorizing the training data than in the conventional single-environment setting. In addition, we find that appropriate regularization on the value network is necessary to improve both training and test performance. To this end, we propose Delayed-Critic Policy Gradient (DCPG), a policy gradient algorithm that implicitly penalizes value estimates by optimizing the value network less frequently with more training data than the policy network. This can be implemented using a single unified network architecture. Furthermore, we introduce a simple self-supervised task that learns the forward and inverse dynamics of environments using a single discriminator, which can be jointly optimized with the value network. Our proposed algorithms significantly improve observational generalization performance and sample efficiency on the Procgen Benchmark.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

36 of 54 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  12. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
24 more
  1. 2020Reinforcement Learning with Augmented Datacited by 468
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  4. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  5. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  6. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  7. 2020Prioritized Level Replaycited by 198
  8. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  9. 2020Phasic Policy Gradientcited by 123
  10. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  11. 2018Generalization and Regularization in DQNcited by 118
  12. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  13. 2019Observational Overfitting in Reinforcement Learningcited by 91
  14. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  15. 2020Deep Reinforcement and InfoMax Learningcited by 79
  16. 2019BackPACK: Packing more into backpropcited by 78
  17. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  18. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  19. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 45
  20. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  21. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 31
  22. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  23. 2017On overfitting and asymptotic bias in batch reinforcement learning with partial observabilitycited by 16
  24. 2017Shallow Updates for Deep Reinforcement Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 5.

8 papers.

  1. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 8
  2. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  3. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  4. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularizationcited by 1
  5. 2025Adaptive Data Exploitation in Deep Reinforcement Learningcited by 1
  6. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learningcited by 1
  7. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  8. 2025Zero-Shot Action Generalization with Limited Observationscited by 0