mlsanityData to 14 Oct 2026

Optimistic critics can empower small actors

arXiv 2506.01016 ↗First posted 1 Jun 2025cs.LG stat.MLcited by 4

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Actor-critic methods have been central to many of the recent advances in deep reinforcement learning. The most common approach is to use symmetric architectures, whereby both actor and critic have the same network topology and number of parameters. However, recent works have argued for the advantages of asymmetric setups, specifically with the use of smaller actors. We perform broad empirical investigations and analyses to better understand the implications of this and find that, in general, smaller actors result in performance degradation and overfit critics. Our analyses suggest poor data collection, due to value underestimation, as one of the main causes for this behavior, and further highlight the crucial role the critic can play in alleviating this pathology. We explore techniques to mitigate the observed value underestimation, which enables further research in asymmetric actor-critic methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 60 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

41 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning29 Jan
  2. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning1 Mar

Other references · 39

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2018DeepMind Control Suitecited by 956
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
27 more
  1. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  3. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  4. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  5. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  6. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  7. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  8. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
  9. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  10. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  11. 2020On Interpretability of Artificial Neural Networks: A Surveycited by 88
  12. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  13. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  14. 2024Visual Whole-Body Control for Legged Loco-Manipulationcited by 62
  15. 2020Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Surveycited by 58
  16. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  17. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  18. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  19. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  20. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 25
  21. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  22. 2022Benchmarking Reinforcement Learning Techniques for Autonomous Navigationcited by 19
  23. 2020WD3: Taming the Estimation Bias in Deep Reinforcement Learningcited by 16
  24. 2024A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismocited by 15
  25. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  26. 2022A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learningcited by 10
  27. 2021Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RLcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 10
  2. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  3. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  4. 2026Augmenting Game AI with Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  2. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  3. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning
  4. 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errors
  5. 2024Value Improved Actor Critic Algorithms
  6. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithms
  7. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
  8. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Exploration