# A survey on multi-objective hyperparameter optimization algorithms for Machine Learning

arXiv: https://arxiv.org/abs/2111.13755 · HTML: https://mlsanity.com/paper/2111.13755/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 188 references that are in the map.
- First posted on arXiv 23 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution](https://mlsanity.com/paper/1804.09081/) (2018)
- [ParamILS: An Automatic Algorithm Configuration Framework](https://mlsanity.com/paper/1401.3492/) (2014)
- [NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search](https://mlsanity.com/paper/2007.10396/) (2020)
- [Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits](https://mlsanity.com/paper/2002.02518/) (2020)
- [AtomNAS: Fine-Grained End-to-End Neural Architecture Search](https://mlsanity.com/paper/1912.09640/) (2019)
- [Hyp-RL : Hyperparameter Optimization by Reinforcement Learning](https://mlsanity.com/paper/1906.11527/) (2019)
- [Replication or exploration? Sequential design for stochastic simulation experiments](https://mlsanity.com/paper/1710.03206/) (2017)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/1609.01051/) (2016)
- [A Bayesian approach to constrained single- and multi-objective optimization](https://mlsanity.com/paper/1510.00503/) (2015)
- [Evolutionary Neural AutoML for Deep Learning](https://mlsanity.com/paper/1902.06827/) (2019)
- [Designing Adaptive Neural Networks for Energy-Constrained Image Classification](https://mlsanity.com/paper/1808.01550/) (2018)
- [Enabling hyperparameter optimization in sequential autoencoders for spiking neural data](https://mlsanity.com/paper/1908.07896/) (2019)
- [Multi-objective Bayesian optimisation with preferences over objectives](https://mlsanity.com/paper/1902.04228/) (2019)
- [HyperPower: Power- and Memory-Constrained Hyper-Parameter Optimization for Neural Networks](https://mlsanity.com/paper/1712.02446/) (2017)
- [Evolving Deep Neural Networks by Multi-objective Particle Swarm Optimization for Image Classification](https://mlsanity.com/paper/1904.09035/) (2019)
- [Towards modular and programmable architecture search](https://mlsanity.com/paper/1909.13404/) (2019)
- [Parameter Optimization and Learning in a Spiking Neural Network for UAV Obstacle Avoidance targeting Neuromorphic Processors](https://mlsanity.com/paper/1910.07960/) (2019)
- [Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization](https://mlsanity.com/paper/1611.02120/) (2016)
- [PABO: Pseudo Agent-Based Multi-Objective Bayesian Hyperparameter Optimization for Efficient Neural Accelerator Design](https://mlsanity.com/paper/1906.08167/) (2019)
- [A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment](https://mlsanity.com/paper/1810.07748/) (2018)
- [GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms](https://mlsanity.com/paper/2012.04201/) (2020)
- [Complexity Theory for Discrete Black-Box Optimization Heuristics](https://mlsanity.com/paper/1801.02037/) (2018)

## Built on by (most cited first, 24 of 24)

- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [LightCTS: A Lightweight Framework for Correlated Time Series Forecasting](https://mlsanity.com/paper/2302.11974/) (2023)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning](https://mlsanity.com/paper/2309.03581/) (2023)
- [Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models](https://mlsanity.com/paper/2307.08175/) (2023)
- [Multi-objective Deep Learning: Taxonomy and Survey of the State of the Art](https://mlsanity.com/paper/2412.01566/) (2024)
- [General Cyclical Training of Neural Networks](https://mlsanity.com/paper/2202.08835/) (2022)
- [Multi-Objective Population Based Training](https://mlsanity.com/paper/2306.01436/) (2023)
- [Beyond transparency: computational reliabilism as an externalist epistemology of algorithms](https://mlsanity.com/paper/2502.20402/) (2025)
- [Mind the Gap: Measuring Generalization Performance Across Multiple Objectives](https://mlsanity.com/paper/2212.04183/) (2022)
- [Dual-stage optimizer for systematic overestimation adjustment applied to multi-objective genetic algorithms for biomarker selection](https://mlsanity.com/paper/2312.16624/) (2023)
- [The Power of the Pareto Front: Balancing Uncertain Rewards for Adaptive Experimentation in scanning probe microscopy](https://mlsanity.com/paper/2504.06525/) (2025)
- [HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services](https://mlsanity.com/paper/2304.10051/) (2023)
- [A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs](https://mlsanity.com/paper/2405.15303/) (2024)
- [Reward driven discovery of the optimal microstructure representations with invariant variational autoencoders](https://mlsanity.com/paper/2510.00243/) (2025)
- [Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation](https://mlsanity.com/paper/2212.06312/) (2022)
- [Damped Gauss Newton Search for Multi Metric Hyperparameter Optimization](https://mlsanity.com/paper/2401.03580/) (2024)
- [Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble](https://mlsanity.com/paper/2402.01379/) (2024)
- [Application of the Brain Drain Optimization Algorithm to the N-Queens Problem](https://mlsanity.com/paper/2504.18953/) (2025)
- [Interpreting Graph Inference with Skyline Explanations](https://mlsanity.com/paper/2505.07635/) (2025)
- [Development of a Multiprocessing Interface Genetic Algorithm for Optimising a Multilayer Perceptron for Disease Prediction](https://mlsanity.com/paper/2506.15694/) (2025)
- [Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization](https://mlsanity.com/paper/2601.03166/) (2026)
- [Leveraging AutoML for Sustainable Deep Learning: A Multi-Objective HPO Approach on Deep Shift Neural Networks](https://mlsanity.com/paper/2606.23208/) (2026)
- [Calibrating Prediction Timeliness Through Multi-Objective Hyperparameter Optimization for Remaining Useful Life Prediction](https://mlsanity.com/paper/2610.01530/) (2026)

## Abstract (arXiv)

Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared that focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.