mlsanityData to 7 Oct 2026

Softmax Deep Double Deterministic Policy Gradients

arXiv 2010.09177 ↗First posted 19 Oct 2020cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

A widely-used actor-critic reinforcement learning algorithm for continuous control, Deep Deterministic Policy Gradients (DDPG), suffers from the overestimation problem, which can negatively affect the performance. Although the state-of-the-art Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm mitigates the overestimation issue, it can lead to a large underestimation bias. In this paper, we propose to use the Boltzmann softmax operator for value function estimation in continuous control. We first theoretically analyze the softmax operator in continuous action space. Then, we uncover an important property of the softmax operator in actor-critic algorithms, i.e., it helps to smooth the optimization landscape, which sheds new light on the benefits of the operator. We also design two new algorithms, Softmax Deep Deterministic Policy Gradients (SD2) and Softmax Deep Double Deterministic Policy Gradients (SD3), by building the softmax operator upon single and double estimators, which can effectively improve the overestimation and underestimation bias. We conduct extensive experiments on challenging continuous control tasks, and results show that SD3 outperforms state-of-the-art methods.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 5; 2023: 7; 2024: 4; 2025: 4; 2026: 4.

31 papers.

Sort:

By area:

  1. 2023Better Training of GFlowNets with Local Credit and Incomplete Trajectoriescited by 83
  2. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  3. 2022Generative Augmented Flow Networkscited by 48
  4. 2023Stochastic Generative Flow Networkscited by 32
  5. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  6. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  7. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  8. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  9. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  10. 2022Revisiting Discrete Soft Actor-Criticcited by 11
  11. 2021Value Activation for Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradientscited by 6
  12. 2024GAS: Generative Auto-bidding with Post-training Searchcited by 6
19 more
  1. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  2. 2023AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systemscited by 5
  3. 2024Random Policy Evaluation Uncovers Policies of Generative Flow Networkscited by 5
  4. 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspectivecited by 4
  5. 2022Effective Multi-User Delay-Constrained Scheduling with Deep Recurrent Reinforcement Learningcited by 3
  6. 2023PAC-Bayesian Soft Actor-Critic Learningcited by 3
  7. 2021Automating Control of Overestimation Bias for Reinforcement Learningcited by 2
  8. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
  9. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Controlcited by 2
  10. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  11. 2025Wasserstein Barycenter Soft Actor-Criticcited by 1
  12. 2026PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspectivecited by 1
  13. 2021Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradientscited by 0
  14. 2022Reducing Learning Difficulties: One-Step Two-Critic Deep Reinforcement Learning for Inverter-based Volt-Var Controlcited by 0
  15. 2022Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methodscited by 0
  16. 2025Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Losscited by 0
  17. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularizationcited by 0
  18. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  19. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0