# Modeling Hierarchical Spaces: A Review and Unified Framework for Surrogate-Based Architecture Design

arXiv: https://arxiv.org/abs/2506.22621 · HTML: https://mlsanity.com/paper/2506.22621/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 174 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 30 of 43)

- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Is Cosine-Similarity of Embeddings Really About Similarity?](https://mlsanity.com/paper/2403.05440/) (2024)
- [On the consistency of supervised learning with missing values](https://mlsanity.com/paper/1902.06931/) (2019)
- [Geodesic Exponential Kernels: When Curvature and Linearity Conflict](https://mlsanity.com/paper/1411.0296/) (2014)
- [Graph Kernels: A Survey](https://mlsanity.com/paper/1904.12218/) (2019)
- [SMT 2.0: A Surrogate Modeling Toolbox with a focus on Hierarchical and Mixed Variables Gaussian Processes](https://mlsanity.com/paper/2305.13998/) (2023)
- [Computational Graph Completion](https://mlsanity.com/paper/2110.10323/) (2021)
- [BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters](https://mlsanity.com/paper/1908.06756/) (2019)
- [Group kernels for Gaussian process metamodels with categorical inputs](https://mlsanity.com/paper/1802.02368/) (2018)
- [Imputation and low-rank estimation with Missing Not At Random data](https://mlsanity.com/paper/1812.11409/) (2018)
- [A mixed-categorical correlation kernel for Gaussian process](https://mlsanity.com/paper/2211.08262/) (2022)
- [The Wasserstein-Fourier Distance for Stationary Time Series](https://mlsanity.com/paper/1912.05509/) (2019)
- [Global Sensitivity Analysis with Dependence Measures](https://mlsanity.com/paper/1311.2483/) (2013)
- [An Empirical Evaluation of Similarity Measures for Time Series Classification](https://mlsanity.com/paper/1401.3973/) (2014)
- [Non-Parametric Representation Learning with Kernels](https://mlsanity.com/paper/2309.02028/) (2023)
- [An Overview of Distance and Similarity Functions for Structured Data](https://mlsanity.com/paper/2002.07420/) (2020)
- [Bayesian optimization of variable-size design space problems](https://mlsanity.com/paper/2003.03300/) (2020)
- [Revisiting Bayesian Optimization in the light of the COCO benchmark](https://mlsanity.com/paper/2103.16649/) (2021)
- [OpenBox: A Python Toolkit for Generalized Black-box Optimization](https://mlsanity.com/paper/2304.13339/) (2023)
- [Continuous surrogate-based optimization algorithms are well-suited for expensive discrete problems](https://mlsanity.com/paper/2011.03431/) (2020)
- [A general mathematical framework for constrained mixed-variable blackbox optimization problems with meta and categorical variables](https://mlsanity.com/paper/2204.00886/) (2022)
- [High-dimensional mixed-categorical Gaussian processes with application to multidisciplinary design optimization for a green aircraft](https://mlsanity.com/paper/2311.06130/) (2023)
- [Codiscovering graphical structure and functional relationships within data: A Gaussian Process framework for connecting the dots](https://mlsanity.com/paper/2311.17007/) (2023)
- [System Architecture Optimization Strategies: Dealing with Expensive Hierarchical Problems](https://mlsanity.com/paper/2502.00838/) (2025)
- [Gaussian process regression with Sliced Wasserstein Weisfeiler-Lehman graph kernels](https://mlsanity.com/paper/2402.03838/) (2024)
- [Bayesian optimization for mixed variables using an adaptive dimension reduction process: applications to aircraft design](https://mlsanity.com/paper/2504.08682/) (2025)

## Built on by (most cited first, 4 of 4)

- [Transfer Learning in Bayesian Optimization for Aircraft Design](https://mlsanity.com/paper/2603.28999/) (2026)
- [From Model-Based Screening to Data-Driven Surrogates: A Multi-Stage Workflow for Exploring Stochastic Agent-Based Models](https://mlsanity.com/paper/2604.03350/) (2026)
- [Efficient multidisciplinary design via Bayesian optimization](https://mlsanity.com/paper/2607.22560/) (2026)
- [Energy-aware frugal Bayesian optimization](https://mlsanity.com/paper/2609.31638/) (2026)

## Abstract (arXiv)

Simulation-based problems involving mixed-variable inputs frequently feature domains that are hierarchical, conditional, heterogeneous, or tree-structured. These characteristics pose challenges for data representation, modeling, and optimization. This paper reviews extensive literature on these structured input spaces and proposes a unified framework that generalizes existing approaches. In this framework, input variables may be continuous, integer, or categorical. A variable is described as meta if its value governs the presence of other decreed variables, enabling the modeling of conditional and hierarchical structures. We further introduce the concept of partially-decreed variables, whose activation depends on contextual conditions. To capture these inter-variable hierarchical relationships, we introduce design space graphs, combining principles from feature modeling and graph theory. This allows the definition of general hierarchical domains suitable for describing complex system architectures. Our framework defines hierarchical distances and kernels to enable surrogate modeling and optimization on hierarchical domains. We demonstrate its effectiveness on complex system design problems, including a neural network and a green-aircraft case study. Our methods are available in the open-source Surrogate Modeling Toolbox (SMT 2.0).