# Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs

arXiv: https://arxiv.org/abs/2509.17998 · HTML: https://mlsanity.com/paper/2509.17998/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Unexpected Improvements to Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2310.20708/) (2023)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Evolution through Large Models](https://mlsanity.com/paper/2206.08896/) (2022)
- [Gaussian Process Kernels for Pattern Discovery and Extrapolation](https://mlsanity.com/paper/1302.4245/) (2013)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations](https://mlsanity.com/paper/1908.01425/) (2019)
- [Recent Advances in Bayesian Optimization](https://mlsanity.com/paper/2206.03301/) (2022)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Automatic Construction and Natural-Language Description of Nonparametric Regression Models](https://mlsanity.com/paper/1402.4304/) (2014)
- [FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch](https://mlsanity.com/paper/2406.04824/) (2024)
- [Bayesian Optimization using Deep Gaussian Processes](https://mlsanity.com/paper/1905.03350/) (2019)
- [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](https://mlsanity.com/paper/2205.14090/) (2022)
- [Kernel Identification Through Transformers](https://mlsanity.com/paper/2106.08185/) (2021)
- [LLM-Enhanced Bayesian Optimization for Efficient Analog Layout Constraint Generation](https://mlsanity.com/paper/2406.05250/) (2024)
- [Diagnosing and fixing common problems in Bayesian optimization for molecule design](https://mlsanity.com/paper/2406.07709/) (2024)
- [Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport](https://mlsanity.com/paper/2210.11836/) (2022)

## Built on by (most cited first, 17 of 17)

- [Unleashing LLMs in Bayesian Optimization: Preference-Guided Framework for Scientific Discovery](https://mlsanity.com/paper/2605.17976/) (2026)
- [LEVI: Stronger Search Architectures Can Substitute for Larger LLMs in Evolutionary Search](https://mlsanity.com/paper/2605.09764/) (2026)
- [LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentation](https://mlsanity.com/paper/2605.22054/) (2026)
- [Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch](https://mlsanity.com/paper/2608.00316/) (2026)
- [ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond](https://mlsanity.com/paper/2608.14354/) (2026)
- [DASH: Decoupled Adaptive Surrogate - Acquisition Harness for Automated Bayesian Optimization](https://mlsanity.com/paper/2608.00641/) (2026)
- [Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization](https://mlsanity.com/paper/2605.20249/) (2026)
- [The Kernel Manifold: A Geometric Approach to Gaussian Process Model Selection](https://mlsanity.com/paper/2601.05371/) (2026)
- [Evidence-Gated LLM Priors for Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2606.01730/) (2026)
- [Reliability-Aware Bayesian Optimization of 1310 nm PCSELs with FDTD Verification](https://mlsanity.com/paper/2607.21772/) (2026)
- [Large Discovery Models: Empirically-grounded Model-Based Open-Ended Search](https://mlsanity.com/paper/2608.15669/) (2026)
- [GRAPE: Gradient Refinement and Progress-Aware Exploitation for Query-Efficient High-Dimensional Bayesian Optimization](https://mlsanity.com/paper/2608.25116/) (2026)
- [Online Adaptive Kernel Mixing for Gaussian Process Decision Making](https://mlsanity.com/paper/2609.19891/) (2026)
- [Robust Bayesian Optimization with Q-Exponential Surrogates](https://mlsanity.com/paper/2609.32775/) (2026)
- [Agentic High-Dimensional Bayesian Optimization with Hypothesis- and Evidence-Guided Search](https://mlsanity.com/paper/2609.34281/) (2026)
- [Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation](https://mlsanity.com/paper/2609.36788/) (2026)
- [Agentic-MatriBO: Capability-Compiled Agentic Bayesian Optimization for Materials and Chemical Design](https://mlsanity.com/paper/2610.17832/) (2026)

## Abstract (arXiv)

The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/richardcsuwandi/cake.