mlsanityData to 7 Oct 2026

Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance

arXiv 2304.11127 ↗First posted 21 Apr 2023cs.LG cs.AIcited by 143

Not on the map. Why?

Abstract (arXiv)

Recent scientific advances require complex experiment design, necessitating the meticulous tuning of many experiment parameters. Tree-structured Parzen estimator (TPE) is a widely used Bayesian optimization method in recent parameter tuning frameworks such as Hyperopt and Optuna. Despite its popularity, the roles of each control parameter in TPE and the algorithm intuition have not been discussed so far. The goal of this paper is to identify the roles of each control parameter and their impacts on parameter tuning based on the ablation studies using diverse benchmark datasets. The recommended setting concluded from the ablation studies is demonstrated to improve the performance of TPE. Our TPE implementation used in this paper is available at https://github.com/nabenabe0928/tpe/tree/single-opt. OptunaHub now provides our standalone TPE implementation at https://hub.optuna.org/samplers/tpe_tutorial/.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

25 of 55 references.

References · 25

  1. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  2. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  3. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  4. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  5. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  6. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  7. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  8. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  9. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  10. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  11. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  12. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 81
13 more
  1. 2022πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimizationcited by 69
  2. 2019Tabular Benchmarks for Joint Architecture and Hyperparameter Optimizationcited by 57
  3. 2018Bayesian Optimization in AlphaGocited by 55
  4. 2022Constrained Efficient Global Optimization of Expensive Black-box Functionscited by 49
  5. 2021BORE: Bayesian Optimization by Density-Ratio Estimationcited by 26
  6. 2022A General Recipe for Likelihood-free Bayesian Optimizationcited by 19
  7. 2020Warm Starting CMA-ES for Hyperparameter Optimizationcited by 18
  8. 2023PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspacescited by 15
  9. 2019Financial Applications of Gaussian Processes and Bayesian Optimizationcited by 12
  10. 2022Batch Bayesian optimisation via density-ratio estimation with guaranteescited by 9
  11. 2022c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimizationcited by 9
  12. 2022Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimatorcited by 7
  13. 2018Rapid Bayesian optimisation for synthesis of short polymer fiber materialscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 29; 2025: 61; 2026: 50.

143 papers.

Sort:

By area:

  1. 2024Large Language Models to Enhance Bayesian Optimizationcited by 127
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  4. 2024CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysiscited by 22
  5. 2024Quantum Kernel Methods under Scrutiny: A Benchmarking Studycited by 22
  6. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 21
  7. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  8. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  9. 2025LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithmscited by 10
  10. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  11. 2024Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Spacecited by 7
  12. 2025Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuningcited by 7
38 more
  1. 2025Towards the Training of Deeper Predictive Coding Neural Networkscited by 7
  2. 2025Decision Tree Induction Through LLMs via Semantically-Aware Evolutioncited by 6
  3. 2024Geometric Neural Network based on Phase Space for BCI-EEG decodingcited by 5
  4. 2024Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detectioncited by 5
  5. 2025Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detectioncited by 5
  6. 2024AutoProteinEngine: A Large Language Model Driven Agent Framework for Multimodal AutoML in Protein Engineeringcited by 4
  7. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  8. 2025ONERA's CRM WBPN database for machine learning activities, related regression challenge and first resultscited by 4
  9. 2026GroupGuard: A Framework for Modeling and Defending Collusive Attacks in Multi-Agent Systemscited by 4
  10. 2026A Fair Evaluation of Graph Foundation Models for Node Property Predictioncited by 4
  11. 2024IEEE BigData 2023 Keystroke Verification Challenge (KVC)cited by 3
  12. 2024HyperbolicLR: Epoch insensitive learning rate schedulercited by 3
  13. 2024TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Modelscited by 3
  14. 2024How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit?cited by 3
  15. 2024Beyond the Boundaries of Proximal Policy Optimizationcited by 3
  16. 2024KVC-onGoing: Keystroke Verification Challengecited by 3
  17. 2025CosmoBench: A Multiscale, Multiview, Multitask Cosmology Benchmark for Geometric Deep Learningcited by 3
  18. 2025MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoEcited by 3
  19. 2025AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agentscited by 3
  20. 2023Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Waitcited by 2
  21. 2024Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarkscited by 2
  22. 2024Chain-structured neural architecture search for financial time series forecastingcited by 2
  23. 2024Fault Detection for agents on power grid topology optimization: A Comprehensive analysiscited by 2
  24. 2024Fine-Grained Alignment in Vision-and-Language Navigation through Bayesian Optimizationcited by 2
  25. 2024Derivation of Closed Form of Expected Improvement for Gaussian Process Trained on Log-Transformed Objectivecited by 2
  26. 2024Unsupervised learning of spatially varying regularization for diffeomorphic image registrationcited by 2
  27. 2025Learning Topology Actions for Power Grid Control: A Graph-Based Soft-Label Imitation Learning Approachcited by 2
  28. 2025Probabilistic Curriculum Learning for Goal-Based Reinforcement Learningcited by 2
  29. 2025Equilibrium Conserving Neural Operators for Super-Resolution Learningcited by 2
  30. 2025Radiometer Calibration using Machine Learningcited by 2
  31. 2025Large Language Model-Empowered Interactive Load Forecastingcited by 2
  32. 2025AGTCNet: A Graph-Temporal Approach for Principled Motor Imagery EEG Classificationcited by 2
  33. 2025Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compressioncited by 2
  34. 2025Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuningcited by 2
  35. 2026FluidFlow: a flow-matching generative model for fluid dynamics surrogates on unstructured meshescited by 2
  36. 2024Compact Model Parameter Extraction via Derivative-Free Optimizationcited by 1
  37. 2024Advancing Machine Learning in Industry 4.0: Benchmark Framework for Rare-event Prediction in Chemical Processescited by 1
  38. 2024A Benchmark on Directed Graph Representation Learning in Hardware Designscited by 1

50 most cited of 143 All 143 (JSON)