mlsanityData to 7 Oct 2026

Zero-Shot AutoML with Pretrained Models

arXiv 2206.08476 ↗First posted 16 Jun 2022cs.LG cs.AI cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Given a new dataset D and a low compute budget, how should we choose a pre-trained model to fine-tune to D, and set the fine-tuning hyperparameters without risking overfitting, particularly if D is small? Here, we extend automated machine learning (AutoML) to best make these choices. Our domain-independent meta-learning approach learns a zero-shot surrogate model which, at test time, allows to select the right deep learning (DL) pipeline (including the pre-trained model and fine-tuning hyperparameters) for a new dataset D given only trivial meta-features describing D such as image resolution or the number of classes. To train this zero-shot model, we collect performance data for many DL pipelines on a large collection of datasets and meta-train on this data to minimize a pairwise ranking objective. We evaluate our approach under the strict time limit of the vision track of the ChaLearn AutoDL challenge benchmark, clearly outperforming all challenge contenders.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 39 of its 91 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

39 of 91 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2017Squeeze-and-Excitation Networkscited by 4,763
  10. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  11. 2017Decoupled Weight Decay Regularizationcited by 1,722
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
27 more
  1. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  2. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  3. 2015Deep Kernel Learningcited by 440
  4. 2018Deep Learning for Classical Japanese Literaturecited by 415
  5. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  6. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  7. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  8. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  9. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  10. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  11. 2018Meta-Learning: A Surveycited by 254
  12. 2019Fast AutoAugmentcited by 243
  13. 2017Generalized Zero-Shot Learning via Synthesized Examplescited by 114
  14. 2018Maximizing acquisition functions for Bayesian optimizationcited by 113
  15. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  16. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  17. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  18. 2019Dataset2Vec: Learning Dataset Meta-Featurescited by 37
  19. 2015ASlib: A Benchmark Library for Algorithm Selectioncited by 36
  20. 2019iCassava 2019 Fine-Grained Visual Categorization Challengecited by 17
  21. 2021Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmarkcited by 16
  22. 2022Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019cited by 16
  23. 2020Extreme Algorithm Selection With Dyadic Feature Representationcited by 11
  24. 2020Practical and sample efficient zero-shot HPOcited by 9
  25. 2019A Meta-Learning Framework for Generalized Zero-Shot Learningcited by 7
  26. 2019RankML: a Meta Learning-Based Approach for Pre-Ranking Machine Learning Pipelinescited by 3
  27. 2020AutoCLINT: The Winning Method in AutoCV Challenge 2019cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 4; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  2. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 17
  3. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearchcited by 12
  4. 2024Large Language Models Synergize with Automated Machine Learningcited by 8
  5. 2023Deep Ranking Ensembles for Hyperparameter Optimizationcited by 5
  6. 2024Transfer Learning for Finetuning Large Language Modelscited by 5
  7. 2025Auto-nnU-Net: Towards Automated Medical Image Segmentationcited by 3
  8. 2023Deep Pipeline Embeddings for AutoMLcited by 2
  9. 2024OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?cited by 2
  10. 2025Estimating Time Series Foundation Model Transferability via In-Context Learningcited by 2
  11. 2026ModelLens: Finding the Best for Your Task from Myriads of Modelscited by 1
  12. 2025How to Select Pre-Trained Code Models for Reuse? A Learning Perspectivecited by 0
1 more
  1. 2025Quickly Tuning Foundation Models for Image Segmentationcited by 0