# Deep Ranking Ensembles for Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2303.15212 · HTML: https://mlsanity.com/paper/2303.15212/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Learning a Deep Listwise Context Model for Ranking Refinement](https://mlsanity.com/paper/1804.05936/) (2018)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [Dataset2Vec: Learning Dataset Meta-Features](https://mlsanity.com/paper/1905.11063/) (2019)
- [Hyp-RL : Hyperparameter Optimization by Reinforcement Learning](https://mlsanity.com/paper/1906.11527/) (2019)
- [Learning to Rank Learning Curves](https://mlsanity.com/paper/2006.03361/) (2020)
- [Zero-Shot AutoML with Pretrained Models](https://mlsanity.com/paper/2206.08476/) (2022)
- [Transferable Neural Processes for Hyperparameter Optimization](https://mlsanity.com/paper/1909.03209/) (2019)
- [Top-Rank Enhanced Listwise Optimization for Statistical Machine Translation](https://mlsanity.com/paper/1707.05438/) (2017)

## Built on by (most cited first, 5 of 5)

- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Model](https://mlsanity.com/paper/2404.11581/) (2024)
- [Ranking over Regression for Bayesian Optimization and Molecule Selection](https://mlsanity.com/paper/2410.09290/) (2024)
- [Deep Pipeline Embeddings for AutoML](https://mlsanity.com/paper/2305.14009/) (2023)
- [MALIBO: Meta-learning for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2307.03565/) (2023)

## Abstract (arXiv)

Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.