mlsanityData to 7 Oct 2026

Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning

arXiv 2205.14557 ↗First posted 29 May 2022cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (DRL) gives the promise that an agent learns good policy from high-dimensional information, whereas representation learning removes irrelevant and redundant information and retains pertinent information. In this work, we demonstrate that the learned representation of the Q-network and its target Q-network should, in theory, satisfy a favorable distinguishable representation property. Specifically, there exists an upper bound on the representation similarity of the value functions of two adjacent time steps in a typical DRL setting. However, through illustrative experiments, we show that the learned DRL agent may violate this property and lead to a sub-optimal policy. Therefore, we propose a simple yet effective regularizer called Policy Evaluation with Easy Regularization on Representation (PEER), which aims to maintain the distinguishable representation property via explicit regularization on internal representations. And we provide the convergence rate guarantee of PEER. Implementing PEER requires only one line of code. Our experiments demonstrate that incorporating PEER into DRL can significantly improve performance and sample efficiency. Comprehensive experiments show that PEER achieves state-of-the-art performance on all 4 environments on PyBullet, 9 out of 12 tasks on DMControl, and 19 out of 26 games on Atari. To the best of our knowledge, PEER is the first work to study the inherent representation property of Q-network and its target. Our code is available at https://sites.google.com/view/peer-cvpr2023/.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 29 May 2022.

Builds on

44 of 68 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
32 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2016Tutorial on Variational Autoencoderscited by 404
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  13. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  14. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  15. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  16. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  17. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  18. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  19. 2019When to use parametric models in reinforcement learning?cited by 118
  20. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  21. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  22. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  23. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  24. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  25. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  26. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  27. 2019The Value Function Polytope in Reinforcement Learningcited by 29
  28. 2013An empirical analysis of dropout in piecewise linear networkscited by 19
  29. 2020WD3: Taming the Estimation Bias in Deep Reinforcement Learningcited by 15
  30. 2017Shallow Updates for Deep Reinforcement Learningcited by 14
  31. 2021MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learningcited by 5
  32. 2017An Upper Bound of the Minimal Dispersion via Delta Coverscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 2.

8 papers.

  1. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  2. 2024Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equationcited by 7
  3. 2024Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignmentcited by 7
  4. 2023Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Controlcited by 5
  5. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  6. 2023Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learningcited by 2
  7. 2024Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learningcited by 2
  8. 2026Expert Behavior Prior Reinforcement Learningcited by 0