mlsanityData to 14 Oct 2026

Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning

arXiv 2311.03711 ↗First posted 7 Nov 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We address the issue of estimation bias in deep reinforcement learning (DRL) by introducing solution mechanisms that include a new, twin TD-regularized actor-critic (TDR) method. It aims at reducing both over and under-estimation errors. With TDR and by combining good DRL improvements, such as distributional learning and long N-step surrogate stage reward (LNSS) method, we show that our new TDR-based actor-critic learning has enabled DRL methods to outperform their respective baselines in challenging environments in DeepMind Control Suite. Furthermore, they elevate TD3 and SAC respectively to a level of performance comparable to that of D4PG (the current SOTA), and they also improve the performance of D4PG to a new SOTA level measured by mean reward, convergence speed, learning success rate, and learning variance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

20 of 31 references.

References · 20

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2018DeepMind Control Suitecited by 956
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Deep Reinforcement Learning that Matterscited by 754
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  9. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  10. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  11. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
8 more
  1. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  2. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 21
  3. 2020Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarkingcited by 19
  4. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 19
  5. 2019Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learningcited by 18
  6. 2018TD-Regularized Actor-Critic Methodscited by 14
  7. 2020The Mean-Squared Error of Double Q-Learningcited by 11
  8. 2016Dynamic Multi-Objectives Optimization with a Changing Number of Objectivescited by 8

Built on by

Not cited yet.