mlsanityData to 7 Oct 2026

Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficiently

arXiv 2507.08053 ↗First posted 10 Jul 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Tree-structured Parzen estimator (TPE) is a versatile hyperparameter optimization (HPO) method supported by popular HPO tools. Since these HPO tools have been developed in line with the trend of deep learning (DL), the problem setups often used in the DL domain have been discussed for TPE such as multi-objective optimization and multi-fidelity optimization. However, the practical applications of HPO are not limited to DL, and black-box combinatorial optimization is actively utilized in some domains, e.g., chemistry and biology. As combinatorial optimization has been an untouched, yet very important, topic in TPE, we propose an efficient combinatorial optimization algorithm for TPE. In this paper, we first generalize the categorical kernel with the numerical kernel in TPE, enabling us to introduce a distance structure to the categorical kernel. Then we discuss modifications for the newly developed kernel to handle a large combinatorial search space. These modifications reduce the time complexity of the kernel calculation with respect to the size of a combinatorial search space. In the experiments using synthetic problems, we verified that our proposed method identifies better solutions with fewer evaluations than the original TPE. Our algorithm is available in Optuna, an open-source framework for HPO.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

22 of 37 references.

References · 22

  1. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  4. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  5. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  7. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 143
  8. 2019Combinatorial Bayesian Optimization using the Graph Cartesian Productcited by 78
  9. 2018Bayesian Optimization of Combinatorial Structurescited by 64
  10. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  11. 2016CMA-ES for Hyperparameter Optimization of Deep Neural Networkscited by 47
  12. 2023Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spacescited by 27
10 more
  1. 2023Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddingscited by 24
  2. 2022On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learningcited by 16
  3. 2023Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimizationcited by 15
  4. 2022c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimizationcited by 9
  5. 2022Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimatorcited by 7
  6. 2024Multi-Fidelity Methods for Optimization: A Surveycited by 5
  7. 2020On the Importance of Domain Model Configuration for Automated Planning Enginescited by 4
  8. 2020Bayesian Variational Optimization for Combinatorial Spacescited by 4
  9. 2023Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Waitcited by 2
  10. 2024Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarkscited by 2

Built on by

Not cited yet.