mlsanityData to 7 Oct 2026

Balancing Value Underestimation and Overestimation with Realistic Actor-Critic

arXiv 2110.09712 ↗First posted 19 Oct 2021cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Model-free deep reinforcement learning (RL) has been successfully applied to challenging continuous control domains. However, poor sample efficiency prevents these methods from being widely used in real-world domains. This paper introduces a novel model-free algorithm, Realistic Actor-Critic(RAC), which can be incorporated with any off-policy RL algorithms to improve sample efficiency. RAC employs Universal Value Function Approximators (UVFA) to simultaneously learn a policy family with the same neural network, each with different trade-offs between underestimation and overestimation. To learn such policies, we introduce uncertainty punished Q-learning, which uses uncertainty from the ensembling of multiple critics to build various confidence-bounds of Q-function. We evaluate RAC on the MuJoCo benchmark, achieving 10x sample efficiency and 25% performance improvement on the most challenging Humanoid environment compared to SAC.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

37 of 50 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2016Deep Exploration via Bootstrapped DQNcited by 518
  11. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  12. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
25 more
  1. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  2. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  3. 2019Self-Supervised Exploration via Disagreementcited by 263
  4. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  5. 2020Training independent subnetworks for robust predictioncited by 177
  6. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  7. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  8. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  9. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  11. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  12. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  13. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  14. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  15. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  16. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  17. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  18. 2019Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitivescited by 30
  19. 2021Ensemble Bootstrapping for Q-Learningcited by 30
  20. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  21. 2020Population-Guided Parallel Policy Search for Reinforcement Learningcited by 20
  22. 2018Learning Awareness Modelscited by 13
  23. 2020SEERL: Sample Efficient Ensemble Reinforcement Learningcited by 11
  24. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  25. 2019Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024ROER: Regularized Optimal Experience Replaycited by 3