mlsanityData to 7 Oct 2026

Bayesian Generational Population-Based Training

arXiv 2207.09405 ↗First posted 19 Jul 2022cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) offers the potential for training generally capable agents that can interact autonomously in the real world. However, one key limitation is the brittleness of RL algorithms to core hyperparameters and network architecture choice. Furthermore, non-stationarities such as evolving training data and increased agent complexity mean that different hyperparameters and architectures may be optimal at different points of training. This motivates AutoRL, a class of methods seeking to automate these design choices. One prominent class of AutoRL methods is Population-Based Training (PBT), which have led to impressive performance in several large scale settings. In this paper, we introduce two new innovations in PBT-style methods. First, we employ trust-region based Bayesian Optimization, enabling full coverage of the high-dimensional mixed hyperparameter search space. Second, we show that using a generational approach, we can also learn both architectures and hyperparameters jointly on-the-fly in a single training run. Leveraging the new highly parallelizable Brax physics engine, we show that these innovations lead to large performance gains, significantly outperforming the tuned baseline while learning entire configurations on the fly. Code is available at https://github.com/xingchenwan/bgpbt.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 53 of its 83 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

53 of 83 references.

References · 53

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2018A Tutorial on Bayesian Optimizationcited by 734
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2017Deep Reinforcement Learning that Matterscited by 727
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  10. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  11. 2017Population Based Training of Neural Networkscited by 351
  12. 2018Quantifying Generalization in Reinforcement Learningcited by 345
38 more
  1. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  2. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  3. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  4. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  5. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  6. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  7. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  8. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  9. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
  10. 2018Meta-Gradient Reinforcement Learningcited by 156
  11. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  12. 2021Replay-Guided Adversarial Environment Designcited by 108
  13. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  14. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  15. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  16. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  17. 2020Discovering Reinforcement Learning Algorithmscited by 75
  18. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  19. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  20. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  21. 2018Bayesian Optimization in AlphaGocited by 55
  22. 2021Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spacescited by 51
  23. 2016Time-Varying Gaussian Process Bandit Optimizationcited by 50
  24. 2021Evolving Reinforcement Learning Algorithmscited by 48
  25. 2021Bootstrapped Meta-Learningcited by 45
  26. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  27. 2020Ready Policy One: World Building Through Active Learningcited by 33
  28. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 31
  29. 2020Optimal Transport Kernels for Sequential and Parallel Neural Architecture Searchcited by 29
  30. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 26
  31. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  32. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16
  33. 2020Deep Reinforcement Learning for Efficient Measurement of Quantum Devicescited by 15
  34. 2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
  35. 2021Faster Improvement Rate Population Based Trainingcited by 12
  36. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 9
  37. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  38. 2021Approximate Neural Architecture Search via Operation Distribution Learningcited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 4; 2025: 5; 2026: 2.

20 papers.

Sort:

  1. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  2. 2022Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterizationcited by 35
  3. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  4. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 24
  5. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  6. 2023AutoRL Hyperparameter Landscapescited by 8
  7. 2023Bayesian Optimisation of Functions on Graphscited by 7
  8. 2024High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategycited by 5
  9. 2024Combining Automated Optimisation of Hyperparameters and Reward Shapecited by 5
  10. 2023Multi-Objective Population Based Trainingcited by 4
  11. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  12. 2024Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learningcited by 2
8 more
  1. 2023Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Searchcited by 1
  2. 2025ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learningcited by 1
  3. 2025Multiple-Frequencies Population-Based Trainingcited by 1
  4. 2025Growing with Experience: Growing Neural Networks in Deep Reinforcement Learningcited by 1
  5. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learningcited by 0
  6. 2025Iterated Population Based Training with Task-Agnostic Restartscited by 0
  7. 2026Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRLcited by 0
  8. 2026Bandits in Prod: Hyperparameter Optimization at Inference Timecited by 0