# Fast Efficient Hyperparameter Tuning for Policy Gradients

arXiv: https://arxiv.org/abs/1902.06583 · HTML: https://mlsanity.com/paper/1902.06583/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2019.

## Built on by (most cited first, 6 of 6)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL](https://mlsanity.com/paper/2106.15883/) (2021)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [A nearly Blackwell-optimal policy gradient method](https://mlsanity.com/paper/2105.13609/) (2021)
- [Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes](https://mlsanity.com/paper/2306.07741/) (2023)

## Abstract (arXiv)

The performance of policy gradient methods is sensitive to hyperparameter settings that must be tuned for any new application. Widely used grid search methods for tuning hyperparameters are sample inefficient and computationally expensive. More advanced methods like Population Based Training that learn optimal schedules for hyperparameters instead of fixed settings can yield better results, but are also sample inefficient and computationally expensive. In this paper, we propose Hyperparameter Optimisation on the Fly (HOOF), a gradient-free algorithm that requires no more than one training run to automatically adapt the hyperparameter that affect the policy update directly through the gradient. The main idea is to use existing trajectories sampled by the policy gradient method to optimise a one-step improvement objective, yielding a sample and computationally efficient algorithm that is easy to implement. Our experimental results across multiple domains and algorithms show that using HOOF to learn these hyperparameter schedules leads to faster learning with improved performance.