# Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance

arXiv: https://arxiv.org/abs/2304.11127 · HTML: https://mlsanity.com/paper/2304.11127/

## Key facts

- Cited by 147 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 25 of 25)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization](https://mlsanity.com/paper/1905.04970/) (2019)
- [Bayesian Optimization in AlphaGo](https://mlsanity.com/paper/1812.06855/) (2018)
- [Constrained Efficient Global Optimization of Expensive Black-box Functions](https://mlsanity.com/paper/2211.00162/) (2022)
- [BORE: Bayesian Optimization by Density-Ratio Estimation](https://mlsanity.com/paper/2102.09009/) (2021)
- [Warm Starting CMA-ES for Hyperparameter Optimization](https://mlsanity.com/paper/2012.06932/) (2020)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces](https://mlsanity.com/paper/2304.10255/) (2023)
- [Financial Applications of Gaussian Processes and Bayesian Optimization](https://mlsanity.com/paper/1903.04841/) (2019)
- [Batch Bayesian optimisation via density-ratio estimation with guarantees](https://mlsanity.com/paper/2209.10715/) (2022)
- [c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization](https://mlsanity.com/paper/2211.14411/) (2022)
- [Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator](https://mlsanity.com/paper/2212.06751/) (2022)
- [Rapid Bayesian optimisation for synthesis of short polymer fiber materials](https://mlsanity.com/paper/1802.05841/) (2018)

## Built on by (most cited first, 30 of 147)

- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [Toto 2.0: Time Series Forecasting Enters the Scaling Era](https://mlsanity.com/paper/2605.20119/) (2026)
- [CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysis](https://mlsanity.com/paper/2409.04290/) (2024)
- [Quantum Kernel Methods under Scrutiny: A Benchmarking Study](https://mlsanity.com/paper/2409.04406/) (2024)
- [Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models](https://mlsanity.com/paper/2505.15406/) (2025)
- [OptunaHub: A Platform for Black-Box Optimization](https://mlsanity.com/paper/2510.02798/) (2025)
- [AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security](https://mlsanity.com/paper/2504.20965/) (2025)
- [LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms](https://mlsanity.com/paper/2505.21034/) (2025)
- [Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Space](https://mlsanity.com/paper/2409.08439/) (2024)
- [Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning](https://mlsanity.com/paper/2502.08056/) (2025)
- [Towards the Training of Deeper Predictive Coding Neural Networks](https://mlsanity.com/paper/2506.23800/) (2025)
- [Decision Tree Induction Through LLMs via Semantically-Aware Evolution](https://mlsanity.com/paper/2503.14217/) (2025)
- [Geometric Neural Network based on Phase Space for BCI-EEG decoding](https://mlsanity.com/paper/2403.05645/) (2024)
- [Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detection](https://mlsanity.com/paper/2409.03141/) (2024)
- [Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detection](https://mlsanity.com/paper/2503.21464/) (2025)
- [Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages](https://mlsanity.com/paper/2503.23542/) (2025)
- [AutoProteinEngine: A Large Language Model Driven Agent Framework for Multimodal AutoML in Protein Engineering](https://mlsanity.com/paper/2411.04440/) (2024)
- [ONERA's CRM WBPN database for machine learning activities, related regression challenge and first results](https://mlsanity.com/paper/2505.06265/) (2025)
- [MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE](https://mlsanity.com/paper/2509.17238/) (2025)
- [AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents](https://mlsanity.com/paper/2510.04257/) (2025)
- [GroupGuard: A Framework for Modeling and Defending Collusive Attacks in Multi-Agent Systems](https://mlsanity.com/paper/2603.13940/) (2026)
- [A Fair Evaluation of Graph Foundation Models for Node Property Prediction](https://mlsanity.com/paper/2606.24509/) (2026)
- [IEEE BigData 2023 Keystroke Verification Challenge (KVC)](https://mlsanity.com/paper/2401.16559/) (2024)
- [HyperbolicLR: Epoch insensitive learning rate scheduler](https://mlsanity.com/paper/2407.15200/) (2024)
- [TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models](https://mlsanity.com/paper/2410.05239/) (2024)
- [How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit?](https://mlsanity.com/paper/2410.15766/) (2024)
- [Beyond the Boundaries of Proximal Policy Optimization](https://mlsanity.com/paper/2411.00666/) (2024)
- [KVC-onGoing: Keystroke Verification Challenge](https://mlsanity.com/paper/2412.20530/) (2024)

## Abstract (arXiv)

Recent scientific advances require complex experiment design, necessitating the meticulous tuning of many experiment parameters. Tree-structured Parzen estimator (TPE) is a widely used Bayesian optimization method in recent parameter tuning frameworks such as Hyperopt and Optuna. Despite its popularity, the roles of each control parameter in TPE and the algorithm intuition have not been discussed so far. The goal of this paper is to identify the roles of each control parameter and their impacts on parameter tuning based on the ablation studies using diverse benchmark datasets. The recommended setting concluded from the ablation studies is demonstrated to improve the performance of TPE. Our TPE implementation used in this paper is available at https://github.com/nabenabe0928/tpe/tree/single-opt. OptunaHub now provides our standalone TPE implementation at https://hub.optuna.org/samplers/tpe_tutorial/.