mlsanityData to 7 Oct 2026

Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL

arXiv 2106.15883 ↗First posted 30 Jun 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for continuous hyperparameters, which severely limits its utility in practice. In this paper we introduce a new (provably) efficient hierarchical approach for optimizing both continuous and categorical variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 56 of its 96 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

56 of 96 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
38 more
  1. 2018Learning Dexterous In-Hand Manipulationcited by 643
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  5. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  6. 2017Population Based Training of Neural Networkscited by 351
  7. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  8. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  9. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  10. 2020Prioritized Level Replaycited by 190
  11. 2018Meta-Gradient Reinforcement Learningcited by 156
  12. 2019Population Based Augmentation: Efficient Learning of Augmentation Policy Schedulescited by 153
  13. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  14. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  15. 2020Phasic Policy Gradientcited by 121
  16. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  17. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  18. 2019Observational Overfitting in Reinforcement Learningcited by 90
  19. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  20. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  21. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  22. 2018Kickstarting Deep Reinforcement Learningcited by 75
  23. 2020Discovering Reinforcement Learning Algorithmscited by 75
  24. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  25. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  26. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  27. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  28. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  29. 2019Emergent Coordination Through Competitioncited by 57
  30. 2018Bayesian Optimization in AlphaGocited by 55
  31. 2016Time-Varying Gaussian Process Bandit Optimizationcited by 50
  32. 2021Evolving Reinforcement Learning Algorithmscited by 48
  33. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 31
  34. 2018Learning to Design RNAcited by 27
  35. 2020Rotation, Translation, and Cropping for Zero-Shot Generalizationcited by 23
  36. 2019A Generalized Framework for Population Based Trainingcited by 22
  37. 2019Knowing The What But Not The Where in Bayesian Optimizationcited by 20
  38. 2019Bayesian Optimization for Iterative Learningcited by 19

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 0; 2024: 0; 2025: 3; 2026: 1.

8 papers.

  1. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  2. 2022Bayesian Generational Population-Based Trainingcited by 20
  3. 2022Event-Triggered Time-Varying Bayesian Optimizationcited by 7
  4. 2022ARLO: A Framework for Automated Reinforcement Learningcited by 2
  5. 2025ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learningcited by 1
  6. 2025Multiple-Frequencies Population-Based Trainingcited by 1
  7. 2025Iterated Population Based Training with Task-Agnostic Restartscited by 0
  8. 2026Bandits in Prod: Hyperparameter Optimization at Inference Timecited by 0