mlsanityData to 7 Oct 2026

Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

arXiv 2001.02811 ↗First posted 9 Jan 2020cs.LG cs.AI cs.SYcited by 80

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), function approximation errors are known to easily lead to the Q-value overestimations, thus greatly reducing policy performance. This paper presents a distributional soft actor-critic (DSAC) algorithm, which is an off-policy RL method for continuous control setting, to improve the policy performance by mitigating Q-value overestimations. We first discover in theory that learning a distribution function of state-action returns can effectively mitigate Q-value overestimations because it is capable of adaptively adjusting the update stepsize of the Q-value function. Then, a distributional soft policy iteration (DSPI) framework is developed by embedding the return distribution function into maximum entropy RL. Finally, we present a deep off-policy actor-critic variant of DSPI, called DSAC, which directly learns a continuous return distribution by keeping the variance of the state-action returns within a reasonable range to address exploding and vanishing gradient problems. We evaluate DSAC on the suite of MuJoCo continuous control tasks, achieving the state-of-the-art performance.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 13; 2023: 12; 2024: 12; 2025: 18; 2026: 16.

80 papers.

Sort:

By area:

  1. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  2. 2022Reachability Constrained Reinforcement Learningcited by 52
  3. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 44
  4. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 25
  5. 2023Distributional Soft Actor-Critic with Three Refinementscited by 22
  6. 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
  7. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  8. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  9. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 16
  10. 2021GMAC: A Distributional Perspective on Actor-Critic Frameworkcited by 15
  11. 2021Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangiancited by 13
  12. 2023DVFO: Learning-Based DVFS for Energy-Efficient Edge-Cloud Collaborative Inferencecited by 13
38 more
  1. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  2. 2022Revisiting Discrete Soft Actor-Criticcited by 11
  3. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokenscited by 9
  4. 2021Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and modelcited by 8
  5. 2021Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learningcited by 6
  6. 2022A Risk-Sensitive Approach to Policy Optimizationcited by 6
  7. 2021Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarioscited by 5
  8. 2021Self-learned Intelligence for Integrated Decision and Control of Automated Vehicles at Signalized Intersectionscited by 5
  9. 2022Diverse Imitation Learning via Self-Organizing Generative Modelscited by 5
  10. 2023Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanismcited by 5
  11. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  12. 2024Conformal Symplectic Optimization for Stable Reinforcement Learningcited by 4
  13. 2025FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learningcited by 4
  14. 2021Steadily Learn to Drive with Virtual Memorycited by 3
  15. 2022Conservative Distributional Reinforcement Learning with Safety Constraintscited by 3
  16. 2022Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Controlcited by 3
  17. 2023Training Multi-layer Neural Networks on Ising Machinecited by 3
  18. 2024Rocket Landing Control with Random Annealing Jump Start Reinforcement Learningcited by 3
  19. 2024How to Choose a Reinforcement-Learning Algorithmcited by 3
  20. 2024Risk-sensitive control as inference with R'enyi divergencecited by 3
  21. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  22. 2021Automating Control of Overestimation Bias for Reinforcement Learningcited by 2
  23. 2023Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversitycited by 2
  24. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  25. 2023Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedbackcited by 2
  26. 2024UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systemscited by 2
  27. 2024Canonical Form of Datatic Description in Control Systemscited by 2
  28. 2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learningcited by 2
  29. 2025Transferable Latent-to-Latent Locomotion Policy for Efficient and Versatile Motion Control of Diverse Legged Robotscited by 2
  30. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  31. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
  32. 2022On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulatorcited by 1
  33. 2023Hybrid of representation learning and reinforcement learning for dynamic and complex robotic motion planningcited by 1
  34. 2024Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategiescited by 1
  35. 2024Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learningcited by 1
  36. 2025CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learningcited by 1
  37. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learningcited by 1
  38. 2026A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metriccited by 1

50 most cited of 80 All 80 (JSON)