Automating Control of Overestimation Bias for Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Overestimation bias control techniques are used by the majority of high-performing off-policy reinforcement learning algorithms. However, most of these techniques rely on pre-defined bias correction policies that are either not flexible enough or require environment-specific tuning of hyperparameters. In this work, we present a general data-driven approach for the automatic selection of bias control hyperparameters. We demonstrate its effectiveness on three algorithms: Truncated Quantile Critics, Weighted Delayed DDPG, and Maxmin Q-learning. The proposed technique eliminates the need for an extensive hyperparameter search. We show that it leads to a significant reduction of the actual number of interactions while preserving the performance.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 31 references that are in the map. First posted on arXiv 26 Oct 2021.
Builds on
16 of 31 references.
References · 16
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
- 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
- 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
- 2019Better Exploration with Optimistic Actor-Criticcited by 88
- 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
- 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
4 more
- 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
- 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
- 2019MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experimentscited by 17
- 2020WD3: Taming the Estimation Bias in Deep Reinforcement Learningcited by 15
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.