# A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs

arXiv: https://arxiv.org/abs/2405.15303 · HTML: https://mlsanity.com/paper/2405.15303/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 30 of 32)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Multi-fidelity Bayesian Optimisation with Continuous Approximations](https://mlsanity.com/paper/1703.06240/) (2017)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations](https://mlsanity.com/paper/1805.12168/) (2018)
- [Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning](https://mlsanity.com/paper/1903.04703/) (2019)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization](https://mlsanity.com/paper/1511.05467/) (2015)
- [Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces](https://mlsanity.com/paper/2109.10964/) (2021)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2009.01721/) (2020)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Multi-objective Bayesian Optimization using Pareto-frontier Entropy](https://mlsanity.com/paper/1906.00127/) (2019)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/1907.01329/) (2019)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)
- [Multi-objective Asynchronous Successive Halving](https://mlsanity.com/paper/2106.12639/) (2021)
- [Multi-Fidelity Multi-Objective Bayesian Optimization: An Output Space Entropy Search Approach](https://mlsanity.com/paper/2011.01542/) (2020)
- [Bayesian Optimization for Iterative Learning](https://mlsanity.com/paper/1909.09593/) (2019)
- [TrimTuner: Efficient Optimization of Machine Learning Jobs in the Cloud via Sub-Sampling](https://mlsanity.com/paper/2011.04726/) (2020)
- [Bayesian Hyperparameter Optimization for Ensemble Learning](https://mlsanity.com/paper/1605.06394/) (2016)
- [Funneled Bayesian Optimization for Design, Tuning and Control of Autonomous Systems](https://mlsanity.com/paper/1610.00366/) (2016)
- [Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach](https://mlsanity.com/paper/2206.12708/) (2022)
- [On the role of Model Uncertainties in Bayesian Optimization](https://mlsanity.com/paper/2301.05983/) (2023)

## Built on by (most cited first, 1 of 1)

- [Converge Then Diversify: Decoupling Convergence and Diversity in Multi-Objective Bayesian Optimisation](https://mlsanity.com/paper/2609.13396/) (2026)

## Abstract (arXiv)

Training machine learning models inherently involves a resource-intensive and noisy iterative learning procedure that allows epoch-wise monitoring of the model performance. However, the insights gained from the iterative learning procedure typically remain underutilized in multi-objective hyperparameter optimization scenarios. Despite the limited research in this area, existing methods commonly identify the trade-offs only at the end of model training, overlooking the fact that trade-offs can emerge at earlier epochs in cases such as overfitting. To bridge this gap, we propose an enhanced multi-objective hyperparameter optimization problem that treats the number of training epochs as a decision variable, rather than merely an auxiliary parameter, to account for trade-offs at an earlier training stage. To solve this problem and accommodate its iterative learning, we then present a trajectory-based multi-objective Bayesian optimization algorithm characterized by two features: 1) a novel acquisition function that captures the improvement along the predictive trajectory of model performances over epochs for any hyperparameter setting and 2) a multi-objective early stopping mechanism that determines when to terminate the training to maximize epoch efficiency. Experiments on synthetic simulations and hyperparameter tuning benchmarks demonstrate that our algorithm can effectively identify the desirable trade-offs while improving tuning efficiency.