mlsanityData to 7 Oct 2026

Target Entropy Annealing for Discrete Soft Actor-Critic

arXiv 2112.02852 ↗First posted 6 Dec 2021cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Soft Actor-Critic (SAC) is considered the state-of-the-art algorithm in continuous action space settings. It uses the maximum entropy framework for efficiency and stability, and applies a heuristic temperature Lagrange term to tune the temperature α, which determines how "soft" the policy should be. It is counter-intuitive that empirical evidence shows SAC does not perform well in discrete domains. In this paper we investigate the possible explanations for this phenomenon and propose Target Entropy Scheduled SAC (TES-SAC), an annealing method for the target entropy parameter applied on SAC. Target entropy is a constant in the temperature Lagrange term and represents the target policy entropy in discrete SAC. We compare our method on Atari 2600 games with different constant target entropy SAC, and analyze on how our scheduling affects SAC.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 11 of its 19 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

11 of 19 references.

References · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  8. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  9. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  10. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  11. 2017RLlib: Abstractions for Distributed Reinforcement Learningcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 1; 2025: 2; 2026: 1.

10 papers.

  1. 2022Revisiting Discrete Soft Actor-Criticcited by 11
  2. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  3. 2023Improving Autonomous Separation Assurance through Distributed Reinforcement Learning with Attention Networkscited by 4
  4. 2023Relational Object-Centric Actor-Criticcited by 4
  5. 2023Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraintcited by 2
  6. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  7. 2026ARROW: Augmented Replay for RObust World modelscited by 1
  8. 2022Entropy Augmented Reinforcement Learningcited by 0
  9. 2023DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Criticcited by 0
  10. 2024Generalizing soft actor-critic algorithms to discrete action spacescited by 0