mlsanityData to 7 Oct 2026

Revisiting Discrete Soft Actor-Critic

arXiv 2209.10081 ↗First posted 21 Sep 2022cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

We study the adaption of Soft Actor-Critic (SAC), which is considered as a state-of-the-art reinforcement learning (RL) algorithm, from continuous action space to discrete action space. We revisit vanilla discrete SAC and provide an in-depth understanding of its Q value underestimation and performance instability issues when applied to discrete settings. We thereby propose Stable Discrete SAC (SDSAC), an algorithm that leverages entropy-penalty and double average Q-learning with Q-clip to address these issues. Extensive experiments on typical benchmarks with discrete action space, including Atari games and a large-scale MOBA game, show the efficacy of our proposed method. Our code is at: https://github.com/coldsummerday/SD-SAC.git.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

28 of 34 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 743
16 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 318
  2. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  3. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  4. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  5. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  6. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  7. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
  8. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  9. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  10. 2022Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learningcited by 27
  11. 2020Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kingscited by 18
  12. 2020Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradientcited by 14
  13. 2020Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Searchcited by 10
  14. 2021Target Entropy Annealing for Discrete Soft Actor-Criticcited by 10
  15. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  16. 2020Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 2; 2026: 1.

11 papers.

  1. 2023Utilizing Reinforcement Learning for de novo Drug Designcited by 7
  2. 2024DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigationcited by 7
  3. 2023Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learningcited by 3
  4. 2024ROER: Regularized Optimal Experience Replaycited by 3
  5. 2023Probabilistic Generative Modeling for Procedural Roundabout Generation for Developing Countriescited by 1
  6. 2023Automatic Truss Design with Reinforcement Learningcited by 0
  7. 2023DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Criticcited by 0
  8. 2024Generalizing soft actor-critic algorithms to discrete action spacescited by 0
  9. 2025Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutionscited by 0
  10. 2025Asynchronous Risk-Aware Multi-Agent Packet Routing for Ultra-Dense LEO Satellite Networkscited by 0
  11. 2026Action Shaping: Policies Absorb What They Can Expresscited by 0