# On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

arXiv: https://arxiv.org/abs/2007.15745 · HTML: https://mlsanity.com/paper/2007.15745/

## Key facts

- Cited by 182 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2020.

## Built on by (most cited first, 30 of 182)

- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Using Large Language Models for Hyperparameter Optimization](https://mlsanity.com/paper/2312.04528/) (2023)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles](https://mlsanity.com/paper/2105.13289/) (2021)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [A Survey of Methods for Automated Algorithm Configuration](https://mlsanity.com/paper/2202.01651/) (2022)
- [A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams](https://mlsanity.com/paper/2104.10529/) (2021)
- [Optimization with Constraint Learning: A Framework and Survey](https://mlsanity.com/paper/2110.02121/) (2021)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)
- [Sampling weights of deep neural networks](https://mlsanity.com/paper/2306.16830/) (2023)
- [Toward Unsupervised Outlier Model Selection](https://mlsanity.com/paper/2211.01834/) (2022)
- [IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective](https://mlsanity.com/paper/2209.08018/) (2022)
- [Machine Learning Towards Intelligent Systems: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2101.03655/) (2021)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [A Transfer Learning and Optimized CNN Based Intrusion Detection System for Internet of Vehicles](https://mlsanity.com/paper/2201.11812/) (2022)
- [An Information Theoretic Evaluation Metric For Strong Unlearning](https://mlsanity.com/paper/2405.17878/) (2024)
- [Multi-Perspective Content Delivery Networks Security Framework Using Optimized Unsupervised Anomaly Detection](https://mlsanity.com/paper/2107.11514/) (2021)
- [An Attention-based ConvLSTM Autoencoder with Dynamic Thresholding for Unsupervised Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2201.09172/) (2022)
- [On TinyML and Cybersecurity: Electric Vehicle Charging Infrastructure Use Case](https://mlsanity.com/paper/2404.16894/) (2024)
- [Search Algorithms for Automated Hyper-Parameter Tuning](https://mlsanity.com/paper/2104.14677/) (2021)
- [REX: Revisiting Budgeted Training with an Improved Schedule](https://mlsanity.com/paper/2107.04197/) (2021)
- [Optimizing YOLOv7 for Semiconductor Defect Detection](https://mlsanity.com/paper/2302.09565/) (2023)
- [Learning to optimize: A tutorial for continuous and mixed-integer optimization](https://mlsanity.com/paper/2405.15251/) (2024)
- [FedTune: Automatic Tuning of Federated Learning Hyper-Parameters from System Perspective](https://mlsanity.com/paper/2110.03061/) (2021)
- [Optimizing Training Trajectories in Variational Autoencoders via Latent Bayesian Optimization Approach](https://mlsanity.com/paper/2207.00128/) (2022)
- [On Leakage in Machine Learning Pipelines](https://mlsanity.com/paper/2311.04179/) (2023)
- [The Power of Training: How Different Neural Network Setups Influence the Energy Demand](https://mlsanity.com/paper/2401.01851/) (2024)

## Abstract (arXiv)

Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.