# Target Entropy Annealing for Discrete Soft Actor-Critic

arXiv: https://arxiv.org/abs/2112.02852 · HTML: https://mlsanity.com/paper/2112.02852/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 11 of 11)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Taming the Noise in Reinforcement Learning via Soft Updates](https://mlsanity.com/paper/1512.08562/) (2015)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [RLlib: Abstractions for Distributed Reinforcement Learning](https://mlsanity.com/paper/1712.09381/) (2017)

## Built on by (most cited first, 10 of 10)

- [Revisiting Discrete Soft Actor-Critic](https://mlsanity.com/paper/2209.10081/) (2022)
- [Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning](https://mlsanity.com/paper/2509.22611/) (2025)
- [Improving Autonomous Separation Assurance through Distributed Reinforcement Learning with Attention Networks](https://mlsanity.com/paper/2308.04958/) (2023)
- [Relational Object-Centric Actor-Critic](https://mlsanity.com/paper/2310.17178/) (2023)
- [Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint](https://mlsanity.com/paper/2303.04356/) (2023)
- [Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning](https://mlsanity.com/paper/2510.08526/) (2025)
- [ARROW: Augmented Replay for RObust World models](https://mlsanity.com/paper/2603.11395/) (2026)
- [Entropy Augmented Reinforcement Learning](https://mlsanity.com/paper/2208.09322/) (2022)
- [DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Critic](https://mlsanity.com/paper/2310.17173/) (2023)
- [Generalizing soft actor-critic algorithms to discrete action spaces](https://mlsanity.com/paper/2407.11044/) (2024)

## Abstract (arXiv)

Soft Actor-Critic (SAC) is considered the state-of-the-art algorithm in continuous action space settings. It uses the maximum entropy framework for efficiency and stability, and applies a heuristic temperature Lagrange term to tune the temperature α, which determines how "soft" the policy should be. It is counter-intuitive that empirical evidence shows SAC does not perform well in discrete domains. In this paper we investigate the possible explanations for this phenomenon and propose Target Entropy Scheduled SAC (TES-SAC), an annealing method for the target entropy parameter applied on SAC. Target entropy is a constant in the temperature Lagrange term and represents the target policy entropy in discrete SAC. We compare our method on Atari 2600 games with different constant target entropy SAC, and analyze on how our scheduling affects SAC.