mlsanityData to 14 Oct 2026

Generalizing soft actor-critic algorithms to discrete action spaces

arXiv 2407.11044 ↗First posted 8 Jul 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

ATARI is a suite of video games used by reinforcement learning (RL) researchers to test the effectiveness of the learning algorithm. Receiving only the raw pixels and the game score, the agent learns to develop sophisticated strategies, even to the comparable level of a professional human games tester. Ideally, we also want an agent requiring very few interactions with the environment. Previous competitive model-free algorithms for the task use the valued-based Rainbow algorithm without any policy head. In this paper, we change it by proposing a practical discrete variant of the soft actor-critic (SAC) algorithm. The new variant enables off-policy learning using policy heads for discrete domains. By incorporating it into the advanced Rainbow variant, i.e., the ``bigger, better, faster'' (BBF), the resulting SAC-BBF improves the previous state-of-the-art interquartile mean (IQM) from 1.045 to 1.088, and it achieves these results using only replay ratio (RR) 2. By using lower RR 2, the training time of SAC-BBF is strictly one-third of the time required for BBF to achieve an IQM of 1.045 using RR 8. As a value of IQM greater than one indicates super-human performance, SAC-BBF is also the only model-free algorithm with a super-human level using only RR 2. The code is publicly available on GitHub at https://github.com/lezhang-thu/bigger-better-faster-SAC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 42 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Auto-Encoding Variational Bayescited by 4,053
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
20 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  5. 2019Model-Based Reinforcement Learning for Ataricited by 502
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  8. 2022Transformers are Sample-Efficient World Modelscited by 328
  9. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  10. 2017Noisy Networks for Explorationcited by 266
  11. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  12. 2021Mastering Atari Games with Limited Datacited by 237
  13. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  14. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  15. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  16. 2019When to use parametric models in reinforcement learning?cited by 118
  17. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  18. 2018Variance reduction properties of the reparameterization trickcited by 41
  19. 2022Revisiting Discrete Soft Actor-Criticcited by 13
  20. 2021Target Entropy Annealing for Discrete Soft Actor-Criticcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dissecting Discrete Soft Actor-Critic: Limitations and Principled Alternatives
  2. 2024PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods
  3. 2022Regularized Soft Actor-Critic for Behavior Transfer Learning
  4. 2025Fusing Rewards and Preferences in Reinforcement Learning
  5. 2026Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning
  6. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  7. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning