# c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2211.14411 · HTML: https://mlsanity.com/paper/2211.14411/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Bayesian Optimization with Unknown Constraints](https://mlsanity.com/paper/1403.5607/) (2014)
- [Constrained Bayesian Optimization with Noisy Experiments](https://mlsanity.com/paper/1706.07094/) (2017)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Scalable Constrained Bayesian Optimization](https://mlsanity.com/paper/2002.08526/) (2020)
- [Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization](https://mlsanity.com/paper/1905.04970/) (2019)
- [Bayesian Optimisation over Multiple Continuous and Categorical Inputs](https://mlsanity.com/paper/1906.08878/) (2019)
- [Bayesian Optimization in AlphaGo](https://mlsanity.com/paper/1812.06855/) (2018)
- [Predictive Entropy Search for Bayesian Optimization with Unknown Constraints](https://mlsanity.com/paper/1502.05312/) (2015)
- [CMA-ES for Hyperparameter Optimization of Deep Neural Networks](https://mlsanity.com/paper/1604.07269/) (2016)
- [Practical Design Space Exploration](https://mlsanity.com/paper/1810.05236/) (2018)
- [Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/1907.01329/) (2019)
- [Bayesian Optimization over Hybrid Spaces](https://mlsanity.com/paper/2106.04682/) (2021)
- [Constrained Bayesian Optimization with Max-Value Entropy Search](https://mlsanity.com/paper/1910.07003/) (2019)

## Built on by (most cited first, 9 of 9)

- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning](https://mlsanity.com/paper/2403.09113/) (2024)
- [Constrained Bayesian Optimization Under Partial Observations: Balanced Improvements and Provable Convergence](https://mlsanity.com/paper/2312.03212/) (2023)
- [Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator](https://mlsanity.com/paper/2212.06751/) (2022)
- [Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks](https://mlsanity.com/paper/2403.01888/) (2024)
- [Derivation of Output Correlation Inferences for Multi-Output (aka Multi-Task) Gaussian Process](https://mlsanity.com/paper/2501.07964/) (2025)
- [Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficiently](https://mlsanity.com/paper/2507.08053/) (2025)
- [Feasible-First Exploration for Constrained ML Deployment Optimization in Crash-Prone Hierarchical Search Spaces](https://mlsanity.com/paper/2604.25073/) (2026)
- [Optuna Constrained Tree-Structured Parzen Estimator Is a Joint Density Generalization of c-TPE](https://mlsanity.com/paper/2606.09889/) (2026)

## Abstract (arXiv)

Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as on memory usage or latency, on top of the performance requirement. In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints. Our proposed extension goes beyond a simple combination of an existing acquisition function and the original TPE, and instead includes modifications that address issues that cause poor performance. We thoroughly analyze these modifications both empirically and theoretically, providing insights into how they effectively overcome these challenges. In the experiments, we demonstrate that c-TPE exhibits the best average rank performance among existing methods with statistical significance on 81 expensive HPO problems with inequality constraints. Due to the lack of baselines, we only discuss the applicability of our method to hard-constrained optimization in Appendix D. The implementation is now available via OptunaHub.