Data to 7 Oct 2026

Decoupling Value and Policy for Generalization in Reinforcement Learning

arXiv 2102.10330 ↗First posted 20 Feb 2021cs.LG cs.AIcited by 81

Not on the map. Why?

Abstract (arXiv)

Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images. However, we argue that more information is needed to accurately estimate the value function than to learn the optimal policy. Consequently, the use of a shared representation for the policy and value function can lead to overfitting. To alleviate this problem, we propose two approaches which are combined to create IDAAC: Invariant Decoupled Advantage Actor-Critic. First, IDAAC decouples the optimization of the policy and value function, using separate networks to model them. Second, it introduces an auxiliary loss which encourages the representation to be invariant to task-irrelevant properties of the environment. IDAAC shows good generalization to unseen environments, achieving a new state-of-the-art on the Procgen benchmark and outperforming popular methods on DeepMind Control tasks with distractors. Our implementation is available at https://github.com/rraileanu/idaac.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 50 of its 60 references that are in the map. First posted on arXiv 20 Feb 2021.

Builds on

50 of 60 references.

References · 50

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2017Proximal Policy Optimization Algorithmscited by 14,441
2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
2015Trust Region Policy Optimizationcited by 2,503
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2019Invariant Risk Minimizationcited by 1,832
2018DeepMind Control Suitecited by 935
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
2020Reinforcement Learning with Augmented Datacited by 452

38 more

2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
2018Quantifying Generalization in Reinforcement Learningcited by 345
2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
2020Decoupling Representation Learning from Reinforcement Learningcited by 263
2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
2020The NetHack Learning Environmentcited by 196
2018Distributed Distributional Deterministic Policy Gradientscited by 195
2020Prioritized Level Replaycited by 190
2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
2017Unsupervised Learning of Disentangled Representations from Videocited by 138
2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
2018Assessing Generalization in Deep Reinforcement Learningcited by 132
2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
2020Phasic Policy Gradientcited by 121
2018Generalization and Regularization in DQNcited by 117
2020Invariant Causal Prediction for Block MDPscited by 104
2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
2019Observational Overfitting in Reinforcement Learningcited by 90
2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
2020Deep Reinforcement and InfoMax Learningcited by 79
2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
2017Towards Generalization and Simplicity in Continuous Controlcited by 65
2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translationcited by 50
2020Interference and Generalization in Temporal Difference Learningcited by 41
2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
2020Rotation, Translation, and Cropping for Zero-Shot Generalizationcited by 23
2020Measuring Visual Generalization in Continuous Control from Pixelscited by 20
2020Instance based Generalization in Reinforcement Learningcited by 17
2020Fast Adaptation via Policy-Dynamics Value Functionscited by 13
2020Reinforcement Learning Generalization with Surprise Minimizationcited by 10
2020Visual Transfer for Reinforcement Learning via Wasserstein Domain Confusioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 18; 2023: 17; 2024: 9; 2025: 17; 2026: 9.

81 papers.

Sort:

By area:

2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30

38 more

2022Hyperbolic Deep Reinforcement Learningcited by 25
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
2023Causal Reinforcement Learning: A Surveycited by 22
2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
2023Explore to Generalize in Zero-Shot RLcited by 20
2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
2022Online Policy Optimization for Robust MDPcited by 19
2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
2022ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representationcited by 11
2023Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learningcited by 11
2022Reinforcement Learning Algorithms: An Overview and Classificationcited by 10
2022Local Feature Swapping for Generalization in Reinforcement Learningcited by 9
2024RELIEF: Reinforcement Learning Empowered Graph Feature Prompt Tuningcited by 9
2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
2025On Generalization Across Environments In Multi-Objective Reinforcement Learningcited by 8
2023DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewardscited by 7
2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agentscited by 7
2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scalecited by 6
2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5
2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
2021Differentiable Architecture Search for Reinforcement Learningcited by 4
2022GriddlyJS: A Web IDE for Reinforcement Learningcited by 4
2023Generalization in Visual Reinforcement Learning with the Reward Sequence Distributioncited by 4
2023RLLTE: Long-Term Evolution Project of Reinforcement Learningcited by 3
2025PanoDreamer: Consistent Text to 360-Degree Scene Generationcited by 3
2025Learning and Editing Universal Graph Prompt Tuning via Reinforcement Learningcited by 3
2022DNA: Proximal Policy Optimization with a Dual Network Architecturecited by 2
2023Adversarial Style Transfer for Robust Policy Optimization in Deep Reinforcement Learningcited by 2

50 most cited of 81 All 81 (JSON)