mlsanityData to 7 Oct 2026

Freeze-Thaw Bayesian Optimization

arXiv 1406.3896 ↗First posted 16 Jun 2014stat.ML cs.LGcited by 86

Not on the map. Why?

Abstract (arXiv)

In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings. Our method uses the partial information gained during the training of a machine learning model in order to decide whether to pause training and start a new model, or resume the training of a previously-considered model. We specifically tailor our method to machine learning problems by developing a novel positive-definite covariance kernel to capture a variety of training curves. Furthermore, we develop a Gaussian process prior that scales gracefully with additional temporal observations. Finally, we provide an information-theoretic framework to automate the decision process. Experiments on several common machine learning models show that our approach is extremely effective in practice.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 12; 2023: 8; 2024: 14; 2025: 10; 2026: 17.

86 papers.

Sort:

By area:

  1. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  2. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  3. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 61
  4. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  5. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  6. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  7. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  8. 2022Learning Curves for Decision Making in Supervised Machine Learning: A Surveycited by 27
  9. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  10. 2021Towards Green Automated Machine Learning: Status Quo and Future Directionscited by 25
  11. 2022AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuningcited by 22
  12. 2021Automatic Termination for Hyperparameter Optimizationcited by 21
38 more
  1. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  2. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  3. 2021Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structurecited by 15
  4. 2022Supervising the Multi-Fidelity Race of Hyperparameter Configurationscited by 15
  5. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  6. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  7. 2024Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selectioncited by 13
  8. 2021Faster Improvement Rate Population Based Trainingcited by 12
  9. 2023Optimizing Hyperparameters with Conformal Quantile Regressioncited by 10
  10. 2021Accelerating Evolutionary Neural Architecture Search via Multi-Fidelity Evaluationcited by 9
  11. 2021RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halvingcited by 9
  12. 2022Amortized Proximal Optimizationcited by 9
  13. 2024Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structurecited by 9
  14. 2026HARBOR: Automated Harness Optimizationcited by 9
  15. 2021Non-smooth Bayesian Optimization in Tuning Problemscited by 8
  16. 2022Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approachcited by 8
  17. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 8
  18. 2025When to retrain a machine learning modelcited by 8
  19. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  20. 2021Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiationcited by 7
  21. 2022TransBO: Hyperparameter Optimization via Two-Phase Transfer Learningcited by 7
  22. 2023Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Singlecited by 7
  23. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  24. 2021Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networkscited by 6
  25. 2022FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimizationcited by 6
  26. 2022Neural Architecture Search for Energy Efficient Always-on Audio Modelscited by 5
  27. 2024Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equationcited by 5
  28. 2021Efficient Hyperparameter Optimization for Physics-based Character Animationcited by 4
  29. 2021Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendationcited by 4
  30. 2021HyperJump: Accelerating HyperBand via Risk Modellingcited by 4
  31. 2021Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizerscited by 4
  32. 2021Cost-Efficient Online Hyperparameter Optimizationcited by 2
  33. 2024NeuroLGP-SM: A Surrogate-assisted Neuroevolution Approach using Linear Genetic Programmingcited by 2
  34. 2024Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolationcited by 2
  35. 2025Information-theoretic Bayesian Optimization: Survey and Tutorialcited by 2
  36. 2025Tune My Adam, Please!cited by 2
  37. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  38. 2025Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuningcited by 2

50 most cited of 86 All 86 (JSON)