# Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces

arXiv: https://arxiv.org/abs/1409.4011 · HTML: https://mlsanity.com/paper/1409.4011/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2014.

## Built on by (most cited first, 16 of 16)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Vanilla Bayesian Optimization Performs Great in High Dimensions](https://mlsanity.com/paper/2402.02229/) (2024)
- [How Powerful are Performance Predictors in Neural Architecture Search?](https://mlsanity.com/paper/2104.01177/) (2021)
- [PFNs4BO: In-Context Learning for Bayesian Optimization](https://mlsanity.com/paper/2305.17535/) (2023)
- [Explaining Hyperparameter Optimization via Partial Dependence Plots](https://mlsanity.com/paper/2111.04820/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)
- [Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/2110.10461/) (2021)
- [Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/2206.01409/) (2022)
- [AutoML for Large Capacity Modeling of Meta's Ranking Systems](https://mlsanity.com/paper/2311.07870/) (2023)
- [GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm](https://mlsanity.com/paper/2206.03291/) (2022)
- [Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization](https://mlsanity.com/paper/2605.20249/) (2026)
- [Modeling All Response Surfaces in One for Conditional Search Spaces](https://mlsanity.com/paper/2501.04260/) (2025)
- [Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space](https://mlsanity.com/paper/2502.09329/) (2025)
- [Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces](https://mlsanity.com/paper/2601.20800/) (2026)

## Abstract (arXiv)

In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.