# Soft Actor-Critic for Discrete Action Settings

arXiv: https://arxiv.org/abs/1910.07207 · HTML: https://mlsanity.com/paper/1910.07207/

## Key facts

- Cited by 174 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 30 of 174)

- [Trajectory balance: Improved credit assignment in GFlowNets](https://mlsanity.com/paper/2201.13259/) (2022)
- [Learning GFlowNets from partial episodes for improved convergence and stability](https://mlsanity.com/paper/2209.12782/) (2022)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [d3rlpy: An Offline Deep Reinforcement Learning Library](https://mlsanity.com/paper/2111.03788/) (2021)
- [Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation](https://mlsanity.com/paper/2110.01219/) (2021)
- [Generative Flow Networks as Entropy-Regularized RL](https://mlsanity.com/paper/2310.12934/) (2023)
- [When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment](https://mlsanity.com/paper/2307.03864/) (2023)
- [Safe Reinforcement Learning via Shielding under Partial Observability](https://mlsanity.com/paper/2204.00755/) (2022)
- [Discrete Probabilistic Inference as Control in Multi-path Environments](https://mlsanity.com/paper/2402.10309/) (2024)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter](https://mlsanity.com/paper/2302.12610/) (2023)
- [LLM-Empowered State Representation for Reinforcement Learning](https://mlsanity.com/paper/2407.13237/) (2024)
- [Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning](https://mlsanity.com/paper/2111.03189/) (2021)
- [Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processes](https://mlsanity.com/paper/2201.12569/) (2022)
- [Enabling AI-Generated Content (AIGC) Services in Wireless Edge Networks](https://mlsanity.com/paper/2301.03220/) (2023)
- [Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2502.04778/) (2025)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [Leveraging Fully Observable Policies for Learning under Partial Observability](https://mlsanity.com/paper/2211.01991/) (2022)
- [Hybrid Multi-agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systems](https://mlsanity.com/paper/2212.07313/) (2022)
- [A General Offline Reinforcement Learning Framework for Interactive Recommendation](https://mlsanity.com/paper/2310.00678/) (2023)
- [Deep Reinforcement Learning-based Large-scale Robot Exploration](https://mlsanity.com/paper/2403.10833/) (2024)
- [AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers](https://mlsanity.com/paper/2411.11188/) (2024)
- [Neural Network Verification in Control](https://mlsanity.com/paper/2110.01388/) (2021)
- [Revisiting Discrete Soft Actor-Critic](https://mlsanity.com/paper/2209.10081/) (2022)
- [EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data](https://mlsanity.com/paper/2406.17768/) (2024)
- [Closing the Planning-Learning Loop with Application to Autonomous Driving](https://mlsanity.com/paper/2101.03834/) (2021)
- [CIMRL: Combining IMitation and Reinforcement Learning for Safe Autonomous Driving](https://mlsanity.com/paper/2406.08878/) (2024)
- [Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers](https://mlsanity.com/paper/2506.00744/) (2025)
- [RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernels](https://mlsanity.com/paper/2310.03912/) (2023)
- [Dream to Drive with Predictive Individual World Model](https://mlsanity.com/paper/2501.16733/) (2025)

## Abstract (arXiv)

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to discrete action settings. We then show that, even without any hyperparameter tuning, it is competitive with the tuned model-free state-of-the-art on a selection of games from the Atari suite.