mlsanityData to 14 Oct 2026

Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges

arXiv 2107.05847 ↗First posted 13 Jul 2021stat.ML cs.LGcited by 145

Not on the map. Why?

Abstract (arXiv)

Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming and unreproducible manual trial-and-error process to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods, e.g., based on resampling error estimation for supervised machine learning, can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization. This work is accompanied by an appendix that contains information on specific software packages in R and Python, as well as information and recommended hyperparameter search spaces for specific learning algorithms. We also provide notebooks that demonstrate concepts from this work as supplementary files.

Key facts. Cited by 145 papers in the map (data through 14 Oct 2026). Builds on 84 of its 275 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

84 of 275 references.

References · 84

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  8. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  9. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  10. 2015Automatic differentiation in machine learning: a surveycited by 855
  11. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  12. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
38 more
  1. 2014OpenML: networked science in machine learningcited by 526
  2. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  3. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  4. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  5. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  6. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 374
  7. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  8. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  9. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  10. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  11. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  12. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  13. 2016Learning to Optimizecited by 195
  14. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 182
  15. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 182
  16. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 179
  17. 2015ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and Rcited by 172
  18. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  19. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  20. 2016GPflow: A Gaussian process library using TensorFlowcited by 140
  21. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 139
  22. 2016COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Settingcited by 138
  23. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 132
  24. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  25. 2014Bayesian Optimization with Unknown Constraintscited by 109
  26. 2018Tunability: Importance of Hyperparameters of Machine Learning Algorithmscited by 99
  27. 2014Freeze-Thaw Bayesian Optimizationcited by 90
  28. 2014ParamILS: An Automatic Algorithm Configuration Frameworkcited by 89
  29. 2015Predictive Entropy Search for Multi-objective Bayesian Optimizationcited by 88
  30. 2018Hyperparameters and Tuning Strategies for Random Forestcited by 84
  31. 2016Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Sciencecited by 75
  32. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  33. 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
  34. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
  35. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 67
  36. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 67
  37. 2020Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDLcited by 67
  38. 2017Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processescited by 66

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 25; 2023: 19; 2024: 39; 2025: 30; 2026: 27.

145 papers.

Sort:

By area:

  1. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  2. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 62
  3. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
  4. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  5. 2021How to avoid machine learning pitfalls: a guide for academic researcherscited by 35
  6. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  7. 2023Visual Tuningcited by 34
  8. 2022A Survey of Methods for Automated Algorithm Configurationcited by 30
  9. 2024AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Modelscited by 30
  10. 2024In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristicscited by 28
  11. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21
  12. 2026Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agentscited by 19
38 more
  1. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearchcited by 15
  2. 2022DeepCAVE: An Interactive Analysis Tool for Automated Machine Learningcited by 14
  3. 2022Transfer Learning based Search Space Design for Hyperparameter Tuningcited by 13
  4. 2021Tune It or Don't Use It: Benchmarking Data-Efficient Image Classificationcited by 12
  5. 2023One Model Many Scores: Using Multiverse Analysis to Prevent Fairness Hacking and Evaluate the Influence of Model Design Decisionscited by 12
  6. 2022Improving Short Text Classification With Augmented Data Using GPT-3cited by 11
  7. 2022Data Augmentation is a Hyperparameter: Cherry-picked Self-Supervision for Unsupervised Anomaly Detection is Creating the Illusion of Successcited by 11
  8. 2023Automated wildlife image classification: An active learning tool for ecological applicationscited by 11
  9. 2024Interpretable Machine Learning for TabPFNcited by 11
  10. 2024This Looks Better than That: Better Interpretable Models with ProtoPNeXtcited by 11
  11. 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 10
  12. 2022HPO X ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysiscited by 10
  13. 2024Hyperparameter Importance Analysis for Multi-Objective AutoMLcited by 9
  14. 2023A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functionscited by 8
  15. 2023Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learningcited by 8
  16. 2022TransBO: Hyperparameter Optimization via Two-Phase Transfer Learningcited by 7
  17. 2024The Power of Training: How Different Neural Network Setups Influence the Energy Demandcited by 7
  18. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  19. 2024Monte Carlo Tree Search based Space Transfer for Black-box Optimizationcited by 7
  20. 2022Enhancing cluster analysis via topological manifold learningcited by 6
  21. 2022Learning Optimal Solutions via an LSTM-Optimization Frameworkcited by 6
  22. 2022AC-Band: A Combinatorial Bandit-Based Approach to Algorithm Configurationcited by 6
  23. 2023Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Modelscited by 6
  24. 2024Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimizationcited by 6
  25. 2025LLM-First Search: Self-Guided Exploration of the Solution Spacecited by 6
  26. 2022General Cyclical Training of Neural Networkscited by 5
  27. 2023Risk-Controlling Model Selection via Guided Bayesian Optimizationcited by 5
  28. 2024Rethinking cluster-conditioned diffusion models for label-free image synthesiscited by 5
  29. 2024A Large-Scale Neutral Comparison Study of Survival Models on Low-Dimensional Datacited by 5
  30. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  31. 2025Overtuning in Hyperparameter Optimizationcited by 5
  32. 2021Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizerscited by 4
  33. 2022Image Classification with Small Datasets: Overview and Benchmarkcited by 4
  34. 2023On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Studycited by 4
  35. 2025Hyperparameter Optimization via Interacting with Probabilistic Circuitscited by 4
  36. 2022A Collection of Quality Diversity Optimization Problems Derived from Hyperparameter Optimization of Machine Learning Modelscited by 3
  37. 2022Position: Tensor Networks are a Valuable Asset for Green AIcited by 3
  38. 2022HARRIS: Hybrid Ranking and Regression Forests for Algorithm Selectioncited by 3

50 most cited of 145 All 145 (JSON)