mlsanityData to 7 Oct 2026

Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview

arXiv 2206.07438 ↗First posted 15 Jun 2022cs.LG stat.MLcited by 37

Not on the map. Why?

Abstract (arXiv)

Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 91 of its 284 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

91 of 284 references.

References · 91

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  12. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
38 more
  1. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  2. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  3. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  4. 2018Robustness May Be at Odds with Accuracycited by 873
  5. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  7. 2016The Mythos of Model Interpretabilitycited by 716
  8. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  9. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  10. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  11. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  12. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  13. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
  14. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  15. 2017The (Un)reliability of saliency methodscited by 331
  16. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  17. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  18. 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 212
  19. 2018A comparative study of fairness-enhancing interventions in machine learningcited by 208
  20. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  21. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  22. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 181
  23. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  24. 2020Minimax Pareto Fairness: A Multi Objective Perspectivecited by 152
  25. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 151
  26. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  27. 2021Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvementcited by 115
  28. 2019Adversarial Examples Are a Natural Consequence of Test Error in Noisecited by 114
  29. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  30. 2018Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examplescited by 93
  31. 2016Measuring Neural Net Robustness with Constraintscited by 90
  32. 2015Predictive Entropy Search for Multi-objective Bayesian Optimizationcited by 88
  33. 2019When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attackscited by 87
  34. 2020Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraintscited by 81
  35. 2019Metric Learning for Adversarial Robustnesscited by 77
  36. 2020NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Searchcited by 75
  37. 2017Preferential Bayesian Optimizationcited by 72
  38. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 7; 2025: 13; 2026: 5.

37 papers.

Sort:

By area:

  1. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  2. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  3. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  4. 2023Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraintscited by 13
  5. 2024Hyperparameter Importance Analysis for Multi-Objective AutoMLcited by 9
  6. 2023Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learningcited by 8
  7. 2023Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Modelscited by 6
  8. 2024Constrained Multi-objective Bayesian Optimization through Optimistic Constraints Estimationcited by 6
  9. 2024Multi-objective Deep Learning: Taxonomy and Survey of the State of the Artcited by 6
  10. 2023Risk-Controlling Model Selection via Guided Bayesian Optimizationcited by 5
  11. 2024X Hacking: The Threat of Misguided AutoMLcited by 5
  12. 2023Multi-Objective Population Based Trainingcited by 4
25 more
  1. 2025The Role of Hyperparameters in Predictive Multiplicitycited by 4
  2. 2026Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Trainingcited by 4
  3. 2022Mind the Gap: Measuring Generalization Performance Across Multiple Objectivescited by 3
  4. 2025MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directionscited by 3
  5. 2023MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimizationcited by 2
  6. 2025The Power of the Pareto Front: Balancing Uncertain Rewards for Adaptive Experimentation in scanning probe microscopycited by 2
  7. 2025Multi-Objective Bilevel Learningcited by 2
  8. 2025Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scorescited by 2
  9. 2023Mithridates: Auditing and Boosting Backdoor Resistance of Machine Learning Pipelinescited by 1
  10. 2023HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Servicescited by 1
  11. 2023Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectivescited by 1
  12. 2024A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offscited by 1
  13. 2025Privilege Scorescited by 1
  14. 2025Reward driven discovery of the optimal microstructure representations with invariant variational autoencoderscited by 1
  15. 2024Damped Gauss Newton Search for Multi Metric Hyperparameter Optimizationcited by 0
  16. 2025Adaptive Resampling with Bootstrap for Noisy Multi-Objective Optimization Problemscited by 0
  17. 2025Multi-Objective Optimization and Hyperparameter Tuning With Desirability Functionscited by 0
  18. 2025Generating Reliable Synthetic Clinical Trial Data: The Role of Hyperparameter Optimization and Domain Constraintscited by 0
  19. 2025Tuning for Trustworthiness -- Balancing Performance and Explanation Consistency in Neural Network Optimizationcited by 0
  20. 2025pared: Model selection using multi-objective optimizationcited by 0
  21. 2025VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scalecited by 0
  22. 2026mlr3mbo: Bayesian Optimization in Rcited by 0
  23. 2026MO-CAPO: Multi-Objective Cost-Aware Prompt Optimizationcited by 0
  24. 2026Converge Then Diversify: Decoupling Convergence and Diversity in Multi-Objective Bayesian Optimisationcited by 0
  25. 2026Calibrating Prediction Timeliness Through Multi-Objective Hyperparameter Optimization for Remaining Useful Life Predictioncited by 0