mlsanityData to 7 Oct 2026

Conservative DDPG -- Pessimistic RL without Ensemble

arXiv 2403.05732 ↗First posted 8 Mar 2024cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

DDPG is hindered by the overestimation bias problem, wherein its Q-estimates tend to overstate the actual Q-values. Traditional solutions to this bias involve ensemble-based methods, which require significant computational resources, or complex log-policy-based approaches, which are difficult to understand and implement. In contrast, we propose a straightforward solution using a Q-target and incorporating a behavioral cloning (BC) loss penalty. This solution, acting as an uncertainty measure, can be easily implemented with minimal code and without the need for an ensemble. Our empirical findings strongly support the superiority of Conservative DDPG over DDPG across various MuJoCo and Bullet tasks. We consistently observe better performance in all evaluated tasks and even competitive or superior performance compared to TD3 and TD7, all achieved with significantly reduced computational requirements.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

14 of 24 references.

References · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  11. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
2 more
  1. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  2. 2019Meta-Q-Learningcited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
  2. 2026UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning
  3. 2022Offline RL Policies Should be Trained to be Adaptive
  4. 2022The Efficacy of Pessimism in Asynchronous Q-Learning
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  6. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  7. 2020Is Pessimism Provably Efficient for Offline RL?
  8. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning