Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Overestimation is a fundamental characteristic of model-free reinforcement learning (MF-RL), arising from the principles of temporal difference learning and the approximation of the Q-function. To address this challenge, we propose a novel moderate target in the Q-function update, formulated as a convex optimization of an overestimated Q-function and its lower bound. Our primary contribution lies in the efficient estimation of this lower bound through the lower expectile of the Q-value distribution conditioned on a state. Notably, our moderate target integrates seamlessly into state-of-the-art (SOTA) MF-RL algorithms, including Deep Deterministic Policy Gradient (DDPG) and Soft Actor Critic (SAC). Experimental results validate the effectiveness of our moderate target in mitigating overestimation bias in DDPG, SAC, and distributional RL algorithms.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 43 references that are in the map. First posted on arXiv 14 Apr 2025.
Builds on
23 of 43 references.
References · 23
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2016OpenAI Gymcited by 1,736
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
11 more
- 2017A Brief Survey of Deep Reinforcement Learningcited by 483
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
- 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
- 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
- 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
- 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
- 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
- 2020Softmax Deep Double Deterministic Policy Gradientscited by 32
- 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 21
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
- 2026Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
- 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
- 2026Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
- 2025ADDQ: Adaptive Distributional Double Q-Learning
- 2024UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
- 2024Bootstrapping Expectiles in Reinforcement Learning
- 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errors