Multiple-Frequencies Population-Based Training
Not on the map. Why?
Abstract (arXiv)
Reinforcement Learning's high sensitivity to hyperparameters is a source of instability and inefficiency, creating significant challenges for practitioners. Hyperparameter Optimization (HPO) algorithms have been developed to address this issue, among them Population-Based Training (PBT) stands out for its ability to generate hyperparameters schedules instead of fixed configurations. PBT trains a population of agents, each with its own hyperparameters, frequently ranking them and replacing the worst performers with mutations of the best agents. These intermediate selection steps can cause PBT to focus on short-term improvements, leading it to get stuck in local optima and eventually fall behind vanilla Random Search over longer timescales. This paper studies how this greediness issue is connected to the choice of evolution frequency, the rate at which the selection is done. We propose Multiple-Frequencies Population-Based Training (MF-PBT), a novel HPO algorithm that addresses greediness by employing sub-populations, each evolving at distinct frequencies. MF-PBT introduces a migration process to transfer information between sub-populations, with an asymmetric design to balance short and long-term optimization. Extensive experiments on the Brax suite demonstrate that MF-PBT improves sample efficiency and long-term performance, even without actually tuning hyperparameters.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 3 Jun 2025.
Builds on
24 of 30 references.
References · 24
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2017Deep Reinforcement Learning that Matterscited by 727
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
- 2015Policy Distillationcited by 414
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
- 2017Population Based Training of Neural Networkscited by 351
- 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
- 2018Meta-Gradient Reinforcement Learningcited by 156
12 more
- 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
- 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
- 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
- 2025Robust Autonomy Emerges from Self-Playcited by 64
- 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
- 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
- 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
- 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 31
- 2022Bayesian Generational Population-Based Trainingcited by 20
- 2021Faster Improvement Rate Population Based Trainingcited by 12
- 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
- 2024Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learningcited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.