# A Self-Tuning Actor-Critic Algorithm

arXiv: https://arxiv.org/abs/2002.12928 · HTML: https://mlsanity.com/paper/2002.12928/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 6 of 6)

- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [Validate on Sim, Detect on Real -- Model Selection for Domain Randomization](https://mlsanity.com/paper/2111.00765/) (2021)
- [A Framework for History-Aware Hyperparameter Optimisation in Reinforcement Learning](https://mlsanity.com/paper/2303.05186/) (2023)
- [Meta-Cognitive Reinforcement Learning with Self-Doubt and Recovery](https://mlsanity.com/paper/2601.20193/) (2026)
- [Adaptive Correlation-Weighted Intrinsic Rewards for Reinforcement Learning](https://mlsanity.com/paper/2602.24081/) (2026)

## Abstract (arXiv)

Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well on a new domain. In this paper, we take a step towards addressing this issue by using metagradients to automatically adapt hyperparameters online by meta-gradient descent (Xu et al., 2018). We apply our algorithm, Self-Tuning Actor-Critic (STAC), to self-tune all the differentiable hyperparameters of an actor-critic loss function, to discover auxiliary tasks, and to improve off-policy learning using a novel leaky V-trace operator. STAC is simple to use, sample efficient and does not require a significant increase in compute. Ablative studies show that the overall performance of STAC improved as we adapt more hyperparameters. When applied to the Arcade Learning Environment (Bellemare et al. 2012), STAC improved the median human normalized score in 200M steps from 243% to 364%. When applied to the DM Control suite (Tassa et al., 2018), STAC improved the mean score in 30M steps from 217 to 389 when learning with features, from 108 to 202 when learning from pixels, and from 195 to 295 in the Real-World Reinforcement Learning Challenge (Dulac-Arnold et al., 2020).