# Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradient

arXiv: https://arxiv.org/abs/2007.01932 · HTML: https://mlsanity.com/paper/2007.01932/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021)
- [Self-Imitation Learning by Planning](https://mlsanity.com/paper/2103.13834/) (2021)
- [Revisiting Discrete Soft Actor-Critic](https://mlsanity.com/paper/2209.10081/) (2022)
- [Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation](https://mlsanity.com/paper/2111.06383/) (2021)
- [Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning](https://mlsanity.com/paper/2602.11779/) (2026)
- [When Maximum Entropy Misleads Policy Optimization](https://mlsanity.com/paper/2506.05615/) (2025)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint](https://mlsanity.com/paper/2303.04356/) (2023)
- [Meta SAC-Lag: Towards Deployable Safe Reinforcement Learning via MetaGradient-based Hyperparameter Tuning](https://mlsanity.com/paper/2408.07962/) (2024)
- [Reinforcement learning with experience replay and adaptation of action dispersion](https://mlsanity.com/paper/2208.00156/) (2022)
- [On Policy Stochasticity in Mutual Information Optimal Control of Linear Systems](https://mlsanity.com/paper/2507.21543/) (2025)
- [DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Critic](https://mlsanity.com/paper/2310.17173/) (2023)
- [Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model](https://mlsanity.com/paper/2604.04986/) (2026)
- [Escaping the Variance Trap: Jacobian-Free Dynamics for Root-Finding Bilevel Optimization](https://mlsanity.com/paper/2606.22433/) (2026)

## Abstract (arXiv)

Exploration-exploitation dilemma has long been a crucial issue in reinforcement learning. In this paper, we propose a new approach to automatically balance between these two. Our method is built upon the Soft Actor-Critic (SAC) algorithm, which uses an "entropy temperature" that balances the original task reward and the policy entropy, and hence controls the trade-off between exploitation and exploration. It is empirically shown that SAC is very sensitive to this hyperparameter, and the follow-up work (SAC-v2), which uses constrained optimization for automatic adjustment, has some limitations. The core of our method, namely Meta-SAC, is to use metagradient along with a novel meta objective to automatically tune the entropy temperature in SAC. We show that Meta-SAC achieves promising performances on several of the Mujoco benchmarking tasks, and outperforms SAC-v2 over 10% in one of the most challenging tasks, humanoid-v2.