mlsanityData to 7 Oct 2026

Hyper-Parameter Optimization: A Review of Algorithms and Applications

arXiv 2003.05689 ↗First posted 12 Mar 2020cs.LG stat.MLcited by 114

Not on the map. Why?

Abstract (arXiv)

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 20; 2023: 21; 2024: 34; 2025: 22; 2026: 5.

114 papers.

Sort:

By area:

  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  2. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  3. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  4. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  5. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  6. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 62
  7. 2021A Survey on Green Deep Learningcited by 52
  8. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  9. 2022A Survey of Methods for Automated Algorithm Configurationcited by 29
  10. 2022Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluationcited by 26
  11. 2022AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuningcited by 22
  12. 2024GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learningcited by 21
38 more
  1. 2022Hyper-parameter tuning of physics-informed neural networks: Application to Helmholtz problemscited by 18
  2. 2023Evolving Scientific Discovery by Unifying Data and Background Knowledge with AI Hilbertcited by 18
  3. 2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 17
  4. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  5. 2021Toward Open-World Electroencephalogram Decoding Via Deep Learning: A Comprehensive Surveycited by 11
  6. 2025Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checkscited by 10
  7. 2022Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learningcited by 9
  8. 2024Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNNcited by 8
  9. 2024The Power of Training: How Different Neural Network Setups Influence the Energy Demandcited by 7
  10. 2022Neural Combinatorial Optimization: a New Player in the Fieldcited by 6
  11. 2022Learning Optimal Solutions via an LSTM-Optimization Frameworkcited by 6
  12. 2023Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?cited by 6
  13. 2024Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applicationscited by 6
  14. 2025Improving LLM-based Global Optimization with Search Space Partitioningcited by 6
  15. 2023Do Not Train It: A Linear Neural Architecture Search of Graph Neural Networkscited by 5
  16. 2024Rethinking cluster-conditioned diffusion models for label-free image synthesiscited by 5
  17. 2024The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimizationcited by 5
  18. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  19. 2022A Comprehensive Survey on Automated Machine Learning for Recommendationscited by 4
  20. 2022Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learningcited by 4
  21. 2023Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networkscited by 4
  22. 2023Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choicecited by 4
  23. 2024Trust Regions for Explanations via Black-Box Probabilistic Certificationcited by 4
  24. 2024Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noisecited by 4
  25. 2024Dynamic Co-Optimization Compiler: Leveraging Multi-Agent Reinforcement Learning for Enhanced DNN Accelerator Performancecited by 4
  26. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Mergingcited by 4
  27. 2021AutoSmart: An Efficient and Automatic Machine Learning framework for Temporal Relational Datacited by 3
  28. 2021Using Sequential Statistical Tests for Efficient Hyperparameter Tuningcited by 3
  29. 2022Batch Layer Normalization, A new normalization layer for CNNs and RNNcited by 3
  30. 2023Can we learn better with hard samples?cited by 3
  31. 2023DP-HyPO: An Adaptive Private Hyperparameter Optimization Frameworkcited by 3
  32. 2024CityGPT: Towards Urban IoT Learning, Analysis and Interaction with Multi-Agent Systemcited by 3
  33. 2021Advancing biological super-resolution microscopy through deep learning: a brief reviewcited by 2
  34. 2022STC-IDS: Spatial-Temporal Correlation Feature Analyzing based Intrusion Detection System for Intelligent Connected Vehiclescited by 2
  35. 2022Deep Gaussian Process-based Multi-fidelity Bayesian Optimization for Simulated Chemical Reactorscited by 2
  36. 2023Scalable End-to-End ML Platforms: from AutoML to Self-servecited by 2
  37. 2023Conductivity Imaging from Internal Measurements with Mixed Least-Squares Deep Neural Networkscited by 2
  38. 2023Label Propagation Techniques for Artifact Detection in Imbalanced Classes using Photoplethysmogram Signalscited by 2

50 most cited of 114 All 114 (JSON)