# Freeze-Thaw Bayesian Optimization

arXiv: https://arxiv.org/abs/1406.3896 · HTML: https://mlsanity.com/paper/1406.3896/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2014.

## Built on by (most cited first, 30 of 90)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [Aioli: A Unified Optimization Framework for Language Model Data Mixing](https://mlsanity.com/paper/2411.05735/) (2024)
- [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2306.12370/) (2023)
- [Learning Curves for Decision Making in Supervised Machine Learning: A Survey](https://mlsanity.com/paper/2201.12150/) (2022)
- [In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2404.16795/) (2024)
- [Towards Green Automated Machine Learning: Status Quo and Future Directions](https://mlsanity.com/paper/2111.05850/) (2021)
- [AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning](https://mlsanity.com/paper/2203.08212/) (2022)
- [Automatic Termination for Hyperparameter Optimization](https://mlsanity.com/paper/2104.08166/) (2021)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [Solving Compositional Reinforcement Learning Problems via Task Reduction](https://mlsanity.com/paper/2103.07607/) (2021)
- [Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure](https://mlsanity.com/paper/2104.11667/) (2021)
- [Supervising the Multi-Fidelity Race of Hyperparameter Configurations](https://mlsanity.com/paper/2202.09774/) (2022)
- [Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection](https://mlsanity.com/paper/2409.15844/) (2024)
- [AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly](https://mlsanity.com/paper/2105.10762/) (2021)
- [Faster Improvement Rate Population Based Training](https://mlsanity.com/paper/2109.13800/) (2021)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [HARBOR: Automated Harness Optimization](https://mlsanity.com/paper/2604.20938/) (2026)
- [Accelerating Evolutionary Neural Architecture Search via Multi-Fidelity Evaluation](https://mlsanity.com/paper/2108.04541/) (2021)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure](https://mlsanity.com/paper/2410.09239/) (2024)
- [RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving](https://mlsanity.com/paper/2108.08019/) (2021)
- [Amortized Proximal Optimization](https://mlsanity.com/paper/2203.00089/) (2022)
- [Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework](https://mlsanity.com/paper/2503.21023/) (2025)
- [Scaling with Collapse: Efficient and Predictable Training of LLM Families](https://mlsanity.com/paper/2509.25087/) (2025)
- [Non-smooth Bayesian Optimization in Tuning Problems](https://mlsanity.com/paper/2109.07563/) (2021)
- [Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach](https://mlsanity.com/paper/2206.12708/) (2022)

## Abstract (arXiv)

In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings. Our method uses the partial information gained during the training of a machine learning model in order to decide whether to pause training and start a new model, or resume the training of a previously-considered model. We specifically tailor our method to machine learning problems by developing a novel positive-definite covariance kernel to capture a variety of training curves. Furthermore, we develop a Gaussian process prior that scales gracefully with additional temporal observations. Finally, we provide an information-theoretic framework to automate the decision process. Experiments on several common machine learning models show that our approach is extremely effective in practice.