mlsanityData to 7 Oct 2026

Efficient Continuous Control with Double Actors and Regularized Critics

arXiv 2106.03050 ↗First posted 6 Jun 2021cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

How to obtain good value estimation is one of the key problems in Reinforcement Learning (RL). Current value estimation methods, such as DDPG and TD3, suffer from unnecessary over- or underestimation bias. In this paper, we explore the potential of double actors, which has been neglected for a long time, for better value function estimation in continuous setting. First, we uncover and demonstrate the bias alleviation property of double actors by building double actors upon single critic and double critics to handle overestimation bias in DDPG and underestimation bias in TD3 respectively. Next, we interestingly find that double actors help improve the exploration ability of the agent. Finally, to mitigate the uncertainty of value estimate from double critics, we further propose to regularize the critic networks under double actors architecture, which gives rise to Double Actors Regularized Critics (DARC) algorithm. Extensive experimental results on challenging continuous control tasks show that DARC significantly outperforms state-of-the-art methods with higher sample efficiency.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

26 of 55 references.

References · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2016OpenAI Gymcited by 1,684
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  11. 2018Distributed Prioritized Experience Replaycited by 260
  12. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
14 more
  1. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  2. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  3. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  4. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  5. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  6. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  7. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  8. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  9. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  10. 2020How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimizationcited by 16
  11. 2019Regularizing Model-Based Planning with Energy-Based Modelscited by 11
  12. 2019On the Reduction of Variance and Overestimation of Deep Q-Learningcited by 4
  13. 2020Unimodal Cyclic Regularization for Training Multimodal Image Registration Networkscited by 3
  14. 2020OPAC: Opportunistic Actor-Criticcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 1; 2023: 2; 2024: 2; 2025: 7; 2026: 4.

20 papers.

Sort:

By area:

  1. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  2. 2021MHER: Model-based Hindsight Experience Replaycited by 14
  3. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  4. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  5. 2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
  6. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  7. 2021Automating Control of Overestimation Bias for Reinforcement Learningcited by 2
  8. 2022Reinforcement Learning for Predicting Traffic Accidentscited by 2
  9. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
  10. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  11. 2025Wasserstein Barycenter Soft Actor-Criticcited by 1
  12. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
8 more
  1. 2023Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learningcited by 0
  2. 2024Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarioscited by 0
  3. 2025Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Losscited by 0
  4. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  5. 2025Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Methodcited by 0
  6. 2025PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explorecited by 0
  7. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularizationcited by 0
  8. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0