mlsanityData to 7 Oct 2026

Deep Ranking Ensembles for Hyperparameter Optimization

arXiv 2303.15212 ↗First posted 27 Mar 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

16 of 41 references.

References · 16

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2017Deep Setscited by 1,394
  3. 2015Deep Kernel Learningcited by 440
  4. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  5. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
  6. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  7. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  8. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  9. 2018Learning a Deep Listwise Context Model for Ranking Refinementcited by 62
  10. 2018Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender Systemcited by 60
  11. 2019Dataset2Vec: Learning Dataset Meta-Featurescited by 37
  12. 2019Hyp-RL : Hyperparameter Optimization by Reinforcement Learningcited by 36
4 more
  1. 2020Learning to Rank Learning Curvescited by 16
  2. 2022Zero-Shot AutoML with Pretrained Modelscited by 13
  3. 2019Transferable Neural Processes for Hyperparameter Optimizationcited by 3
  4. 2017Top-Rank Enhanced Listwise Optimization for Statistical Machine Translationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 17
  2. 2024E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Modelcited by 9
  3. 2024Ranking over Regression for Bayesian Optimization and Molecule Selectioncited by 4
  4. 2023Deep Pipeline Embeddings for AutoMLcited by 2
  5. 2023MALIBO: Meta-learning for Likelihood-free Bayesian Optimizationcited by 2