# Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies

arXiv: https://arxiv.org/abs/2006.07554 · HTML: https://mlsanity.com/paper/2006.07554/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2020.

## Built on by (most cited first, 11 of 11)

- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Bayesian Generational Population-Based Training](https://mlsanity.com/paper/2207.09405/) (2022)
- [No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL](https://mlsanity.com/paper/2205.08716/) (2022)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [Episodic Policy Gradient Training](https://mlsanity.com/paper/2112.01853/) (2021)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2501.15129/) (2025)
- [Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning](https://mlsanity.com/paper/2112.08094/) (2021)
- [BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization](https://mlsanity.com/paper/2308.01207/) (2023)

## Abstract (arXiv)

Off-policy learning algorithms have been known to be sensitive to the choice of hyper-parameters. However, unlike near on-policy algorithms for which hyper-parameters could be optimized via e.g. meta-gradients, similar techniques could not be straightforwardly applied to off-policy learning. In this work, we propose a framework which entails the application of Evolutionary Strategies to online hyper-parameter tuning in off-policy learning. Our formulation draws close connections to meta-gradients and leverages the strengths of black-box optimization with relatively low-dimensional search spaces. We show that our method outperforms state-of-the-art off-policy learning baselines with static hyper-parameters and recent prior work over a wide range of continuous control benchmarks.