mlsanityData to 14 Oct 2026

Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How

arXiv 2306.03828 ↗First posted 6 Jun 2023cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset. In this paper, we propose a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pretrained models with multiple hyperparameter configurations on a series of datasets. To this aim, we evaluated over 20k hyperparameter configurations for finetuning 24 pretrained image classification models on 87 datasets to generate a large-scale meta-dataset. We meta-learn a multi-fidelity performance predictor on the learning curves of this meta-dataset and use it for fast hyperparameter optimization on new datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for a new dataset together with its optimal hyperparameters.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 46 of its 72 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

46 of 72 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2014How transferable are features in deep neural networks?cited by 1,225
  7. 2021Robust fine-tuning of zero-shot modelscited by 797
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  9. 2019A Closer Look at Few-shot Classificationcited by 684
  10. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  11. 2017Deep Layer Aggregationcited by 436
  12. 2021XCiT: Cross-Covariance Image Transformerscited by 338
34 more
  1. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  2. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
  3. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  4. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  5. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
  6. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  7. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
  8. 2019Transferability and Hardness of Supervised Classification Taskscited by 127
  9. 2018Maximizing acquisition functions for Bayesian optimizationcited by 114
  10. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  11. 2016Entity Embeddings of Categorical Variablescited by 87
  12. 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
  13. 2019DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networkscited by 80
  14. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  15. 2021Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learningcited by 72
  16. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
  17. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  18. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 54
  19. 2021Scalable Diverse Model Selection for Accessible Transfer Learningcited by 40
  20. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 39
  21. 2019Dataset2Vec: Learning Dataset Meta-Featurescited by 38
  22. 2021Zoo-Tuning: Adaptive Transfer from a Zoo of Modelscited by 35
  23. 2023Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classificationcited by 30
  24. 2021Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubscited by 28
  25. 2021TransTailor: Pruning the Pre-trained Model for Improved Transfer Learningcited by 25
  26. 2023Scaling Laws for Hyperparameter Optimizationcited by 16
  27. 2020Bi-tuning of Pre-trained Representationscited by 15
  28. 2022Supervising the Multi-Fidelity Race of Hyperparameter Configurationscited by 15
  29. 2022Zero-Shot AutoML with Pretrained Modelscited by 14
  30. 2022Hyper-Representations for Pre-Training and Transfer Learningcited by 12
  31. 2020Practical and sample efficient zero-shot HPOcited by 9
  32. 2022Hub-Pathway: Transfer Learning from A Hub of Pre-trained Modelscited by 5
  33. 2023Deep Ranking Ensembles for Hyperparameter Optimizationcited by 5
  34. 2023Deep Pipeline Embeddings for AutoMLcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 6; 2026: 3.

18 papers.

Sort:

By area:

  1. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  2. 2024AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Modelscited by 30
  3. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearchcited by 15
  4. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  5. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 7
  6. 2024Transfer Learning for Finetuning Large Language Modelscited by 5
  7. 2026Automatic Configuration of LLM Post-Training Pipelinescited by 3
  8. 2024Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolationcited by 2
  9. 2024OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?cited by 2
  10. 2024Ensembling Finetuned Language Models for Text Classificationcited by 2
  11. 2025Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuningcited by 2
  12. 2025Dynamic Priors in Bayesian Optimization for Hyperparameter Optimizationcited by 2
6 more
  1. 2024Regularized Neural Ensemblerscited by 1
  2. 2025Benchmarking Image Embeddings for E-Commerce: Evaluating Off-the Shelf Foundation Models, Fine-Tuning Strategies and Practical Trade-offscited by 1
  3. 2025In-Context Decision Making for Optimizing Complex AutoML Pipelinescited by 0
  4. 2025Quickly Tuning Foundation Models for Image Segmentationcited by 0
  5. 2025Implicit Modeling for Transferability Estimation of Vision Foundation Modelscited by 0
  6. 2026TabMF: Learning Dataset Representations with Tabular Foundation Modelscited by 0