mlsanityData to 14 Oct 2026

Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning

arXiv 2510.21379 ↗First posted 24 Oct 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper, we address the problem of cost-sensitive hyperparameter optimization (HPO) built upon freeze-thaw Bayesian optimization (BO). Specifically, we assume a scenario where users want to early-stop the HPO process when the expected performance improvement is not satisfactory with respect to the additional computational cost. Motivated by this scenario, we introduce utility in the freeze-thaw framework, a function describing the trade-off between the cost and performance that can be estimated from the user's preference data. This utility function, combined with our novel acquisition function and stopping criterion, allows us to dynamically continue training the configuration that we expect to maximally improve the utility in the future, and also automatically stop the HPO process around the maximum utility. Further, we improve the sample efficiency of existing freeze-thaw methods with transfer learning to develop a specialized surrogate model for the cost-sensitive HPO problem. We validate our algorithm on established multi-fidelity HPO benchmarks and show that it outperforms all the previous freeze-thaw BO and transfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance. Our code is publicly available at https://github.com/db-Lee/CFBO.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

45 of 67 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  9. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  10. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  11. 2015Deep Kernel Learningcited by 445
  12. 2021Transformers Can Do Bayesian Inferencecited by 360
33 more
  1. 2018Neural Processescited by 311
  2. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  3. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  4. 2018A System for Massively Parallel Hyperparameter Tuningcited by 202
  5. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 132
  6. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  7. 2017Multi-fidelity Bayesian Optimisation with Continuous Approximationscited by 121
  8. 2019Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuningcited by 93
  9. 2014Freeze-Thaw Bayesian Optimizationcited by 90
  10. 2016Multi-Information Source Optimizationcited by 84
  11. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 83
  12. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 73
  13. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
  14. 2022πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimizationcited by 70
  15. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 67
  16. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  17. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  18. 2020Cost-aware Bayesian Optimizationcited by 42
  19. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  20. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  21. 2020Using a thousand optimization tasks to learn hyperparameter search strategiescited by 29
  22. 2023Transfer Learning for Bayesian Optimization: A Surveycited by 27
  23. 2022Roboflow 100: A Rich, Multi-Domain Object Detection Benchmarkcited by 26
  24. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  25. 2016The Multi-fidelity Multi-armed Banditcited by 22
  26. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 18
  27. 2020Learning to Rank Learning Curvescited by 16
  28. 2023Scaling Laws for Hyperparameter Optimizationcited by 16
  29. 2024Cost-aware Bayesian Optimization via the Pandora's Box Gittins Indexcited by 16
  30. 2022Supervising the Multi-Fidelity Race of Hyperparameter Configurationscited by 15
  31. 2019Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settingscited by 12
  32. 2024Stopping Bayesian Optimization with Probabilistic Regret Boundscited by 11
  33. 2019Cost-aware Multi-objective Bayesian optimisationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Modelscited by 2
  2. 2026The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMscited by 0