mlsanityData to 14 Oct 2026

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

arXiv 1603.06560 ↗First posted 21 Mar 2016cs.LG stat.MLcited by 699

Not on the map. Why?

Cited by 23 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Key facts. Cited by 699 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 131; 2022: 118; 2023: 115; 2024: 95; 2025: 103; 2026: 136.

699 papers.

Sort:

By area:

  1. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  2. 2021Advances in Neural Renderingcited by 365
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  4. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  5. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 230
  6. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  7. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  8. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
  9. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  10. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  11. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 147
  12. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
38 more
  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  2. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  3. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  4. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 94
  5. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 90
  6. 2021HyperMorph: Amortized Hyperparameter Learning for Image Registrationcited by 85
  7. 2022HyperImpute: Generalized Iterative Imputation with Automatic Model Selectioncited by 83
  8. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  9. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  10. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  11. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  12. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  13. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 71
  14. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  15. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  16. 2025InverseBench: Benchmarking Plug-and-Play Diffusion Priors for Inverse Problems in Physical Sciencescited by 68
  17. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  18. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 62
  19. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  20. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  21. 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
  22. 2021Misspecified Gaussian Process Bandit Optimizationcited by 55
  23. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  24. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  25. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  26. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 48
  27. 2021OpenBox: A Generalized Black-box Optimization Servicecited by 46
  28. 2022FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learningcited by 43
  29. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
  30. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  31. 2022Automated Dynamic Algorithm Configurationcited by 39
  32. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 39
  33. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  34. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  35. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  36. 2022Learning the Effect of Registration Hyperparameters with HyperMorphcited by 35
  37. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35
  38. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35

50 most cited of 699 All 699 (JSON)