mlsanityData to 14 Oct 2026

Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equation

arXiv 2404.12754 ↗First posted 19 Apr 2024cs.LG cs.AIcited by 7

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Representation rank is an important concept for understanding the role of Neural Networks (NNs) in Deep Reinforcement learning (DRL), which measures the expressive capacity of value networks. Existing studies focus on unboundedly maximizing this rank; nevertheless, that approach would introduce overly complex models in the learning, thus undermining performance. Hence, fine-tuning representation rank presents a challenging and crucial optimization problem. To address this issue, we find a guiding principle for adaptive control of the representation rank. We employ the Bellman equation as a theoretical foundation and derive an upper bound on the cosine similarity of consecutive state-action pairs representations of value networks. We then leverage this upper bound to propose a novel regularizer, namely BEllman Equation-based automatic rank Regularizer (BEER). This regularizer adaptively regularizes the representation rank, thus improving the DRL agent's performance. We first validate the effectiveness of automatic control of rank on illustrative experiments. Then, we scale up BEER to complex continuous control tasks by combining it with the deterministic policy gradient method. Among 12 challenging DeepMind control tasks, BEER outperforms the baselines by a large margin. Besides, BEER demonstrates significant advantages in Q-value approximation. Our code is available at https://github.com/sweetice/BEER-ICLR2024.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 46 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

37 of 46 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2015Continuous control with deep reinforcement learningcited by 3,407
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
25 more
  1. 2017Deep Reinforcement Learning that Matterscited by 754
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  3. 2020Reinforcement Learning with Augmented Datacited by 468
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  5. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  6. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  10. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  11. 2023Understanding plasticity in neural networkscited by 174
  12. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  13. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  14. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  15. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  16. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  17. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
  18. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  19. 2021Leveraging Good Representations in Linear Contextual Banditscited by 29
  20. 2021Learning Dynamics Models for Model Predictive Agentscited by 22
  21. 2021Reinforcement Learning in Linear MDPs: Constant Regret and Representation Selectioncited by 18
  22. 2020WD3: Taming the Estimation Bias in Deep Reinforcement Learningcited by 16
  23. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RLcited by 12
  24. 2022Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learningcited by 8
  25. 2023Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 2.

7 papers.

  1. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  2. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  3. 2024Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignmentcited by 8
  4. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  5. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  6. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  7. 2026Sustaining Plasticity via Learnable Wavelet Activations in Continual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimal Policy Sparsification and Low Rank Decomposition for Deep Reinforcement Learning
  2. 2025Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
  3. 2022An Empirical Study of Implicit Regularization in Deep Offline RL
  4. 2025High-order Regularization for Machine Learning and Learning-based Control
  5. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  6. 2024Decoupling regularization from the action space
  7. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularization
  8. 2024The Role of Deep Learning Regularizations on Actors in Offline RL