# Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview

arXiv: https://arxiv.org/abs/2206.07438 · HTML: https://mlsanity.com/paper/2206.07438/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 91 of its 284 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 30 of 91)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)

## Built on by (most cited first, 30 of 37)

- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [Position: A Call to Action for a Human-Centered AutoML Paradigm](https://mlsanity.com/paper/2406.03348/) (2024)
- [Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints](https://mlsanity.com/paper/2311.08675/) (2023)
- [Hyperparameter Importance Analysis for Multi-Objective AutoML](https://mlsanity.com/paper/2405.07640/) (2024)
- [Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning](https://mlsanity.com/paper/2309.03581/) (2023)
- [Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models](https://mlsanity.com/paper/2307.08175/) (2023)
- [Constrained Multi-objective Bayesian Optimization through Optimistic Constraints Estimation](https://mlsanity.com/paper/2411.03641/) (2024)
- [Multi-objective Deep Learning: Taxonomy and Survey of the State of the Art](https://mlsanity.com/paper/2412.01566/) (2024)
- [Risk-Controlling Model Selection via Guided Bayesian Optimization](https://mlsanity.com/paper/2312.01692/) (2023)
- [X Hacking: The Threat of Misguided AutoML](https://mlsanity.com/paper/2401.08513/) (2024)
- [Multi-Objective Population Based Training](https://mlsanity.com/paper/2306.01436/) (2023)
- [The Role of Hyperparameters in Predictive Multiplicity](https://mlsanity.com/paper/2503.13506/) (2025)
- [Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training](https://mlsanity.com/paper/2606.21090/) (2026)
- [Mind the Gap: Measuring Generalization Performance Across Multiple Objectives](https://mlsanity.com/paper/2212.04183/) (2022)
- [MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions](https://mlsanity.com/paper/2510.20872/) (2025)
- [MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization](https://mlsanity.com/paper/2305.04502/) (2023)
- [The Power of the Pareto Front: Balancing Uncertain Rewards for Adaptive Experimentation in scanning probe microscopy](https://mlsanity.com/paper/2504.06525/) (2025)
- [Multi-Objective Bilevel Learning](https://mlsanity.com/paper/2511.07824/) (2025)
- [Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores](https://mlsanity.com/paper/2512.01080/) (2025)
- [Mithridates: Auditing and Boosting Backdoor Resistance of Machine Learning Pipelines](https://mlsanity.com/paper/2302.04977/) (2023)
- [HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services](https://mlsanity.com/paper/2304.10051/) (2023)
- [Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives](https://mlsanity.com/paper/2309.14936/) (2023)
- [A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs](https://mlsanity.com/paper/2405.15303/) (2024)
- [Privilege Scores](https://mlsanity.com/paper/2502.01211/) (2025)
- [Reward driven discovery of the optimal microstructure representations with invariant variational autoencoders](https://mlsanity.com/paper/2510.00243/) (2025)
- [MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization](https://mlsanity.com/paper/2605.18869/) (2026)
- [Damped Gauss Newton Search for Multi Metric Hyperparameter Optimization](https://mlsanity.com/paper/2401.03580/) (2024)
- [Adaptive Resampling with Bootstrap for Noisy Multi-Objective Optimization Problems](https://mlsanity.com/paper/2503.21495/) (2025)
- [Multi-Objective Optimization and Hyperparameter Tuning With Desirability Functions](https://mlsanity.com/paper/2503.23595/) (2025)

## Abstract (arXiv)

Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness.