# Obeying the Order: Introducing Ordered Transfer Hyperparameter Optimisation

arXiv: https://arxiv.org/abs/2306.16916 · HTML: https://mlsanity.com/paper/2306.16916/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning](https://mlsanity.com/paper/1909.12552/) (2019)
- [Transfer Learning for Bayesian Optimization: A Survey](https://mlsanity.com/paper/2302.05927/) (2023)
- [Practical Transfer Learning for Bayesian Optimization](https://mlsanity.com/paper/1802.02219/) (2018)
- [BORE: Bayesian Optimization by Density-Ratio Estimation](https://mlsanity.com/paper/2102.09009/) (2021)
- [Automatic Termination for Hyperparameter Optimization](https://mlsanity.com/paper/2104.08166/) (2021)
- [Hyperparameter Transfer Learning with Adaptive Complexity](https://mlsanity.com/paper/2102.12810/) (2021)
- [Hyperparameter Transfer Across Developer Adjustments](https://mlsanity.com/paper/2010.13117/) (2020)
- [A resource-efficient method for repeated HPO and NAS problems](https://mlsanity.com/paper/2103.16111/) (2021)

## Built on by (most cited first, 1 of 1)

- [Hyperparameter Selection in Continual Learning](https://mlsanity.com/paper/2404.06466/) (2024)

## Abstract (arXiv)

We introduce ordered transfer hyperparameter optimisation (OTHPO), a version of transfer learning for hyperparameter optimisation (HPO) where the tasks follow a sequential order. Unlike for state-of-the-art transfer HPO, the assumption is that each task is most correlated to those immediately before it. This matches many deployed settings, where hyperparameters are retuned as more data is collected; for instance tuning a sequence of movie recommendation systems as more movies and ratings are added. We propose a formal definition, outline the differences to related problems and propose a basic OTHPO method that outperforms state-of-the-art transfer HPO. We empirically show the importance of taking order into account using ten benchmarks. The benchmarks are in the setting of gradually accumulating data, and span XGBoost, random forest, approximate k-nearest neighbor, elastic net, support vector machines and a separate real-world motivated optimisation problem. We open source the benchmarks to foster future research on ordered transfer HPO.