# RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernels

arXiv: https://arxiv.org/abs/2310.03912 · HTML: https://mlsanity.com/paper/2310.03912/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization](https://mlsanity.com/paper/1904.02642/) (2019)
- [Boltzmann Exploration Done Right](https://mlsanity.com/paper/1705.10257/) (2017)
- [Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables](https://mlsanity.com/paper/1910.01688/) (2019)
- [Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization](https://mlsanity.com/paper/2106.04335/) (2021)
- [Skip Training for Multi-Agent Reinforcement Learning Controller for Industrial Wave Energy Converters](https://mlsanity.com/paper/2209.05656/) (2022)

## Built on by (most cited first, 11 of 11)

- [A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences](https://mlsanity.com/paper/2406.04739/) (2024)
- [BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL](https://mlsanity.com/paper/2505.21974/) (2025)
- [N-Critics: Self-Refinement of Large Language Models with Ensemble of Critics](https://mlsanity.com/paper/2310.18679/) (2023)
- [Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustness](https://mlsanity.com/paper/2310.18626/) (2023)
- [EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization](https://mlsanity.com/paper/2411.00171/) (2024)
- [Carbon Footprint Reduction for Sustainable Data Centers in Real-Time](https://mlsanity.com/paper/2403.14092/) (2024)
- [Robustness and Visual Explanation for Black Box Image, Video, and ECG Signal Classification with Reinforcement Learning](https://mlsanity.com/paper/2403.18985/) (2024)
- [Function Approximation for Reinforcement Learning Controller for Energy from Spread Waves](https://mlsanity.com/paper/2404.10991/) (2024)
- [A Configurable Pythonic Data Center Model for Sustainable Cooling and ML Integration](https://mlsanity.com/paper/2404.12498/) (2024)
- [NeuroPareto: Calibrated Acquisition for Costly Many-Goal Search in Vast Parameter Spaces](https://mlsanity.com/paper/2602.03901/) (2026)
- [Sustainability of Data Center Digital Twins with Reinforcement Learning](https://mlsanity.com/paper/2404.10786/) (2024)

## Abstract (arXiv)

Bayesian Optimization (BO), guided by Gaussian process (GP) surrogates, has proven to be an invaluable technique for efficient, high-dimensional, black-box optimization, a critical problem inherent to many applications such as industrial design and scientific computing. Recent contributions have introduced reinforcement learning (RL) to improve the optimization performance on both single function optimization and textitfew-shot multi-objective optimization. However, even few-shot techniques fail to exploit similarities shared between closely related objectives. In this paper, we combine recent developments in Deep Kernel Learning (DKL) and attention-based Transformer models to improve the modeling powers of GP surrogates with meta-learning. We propose a novel method for improving meta-learning BO surrogates by incorporating attention mechanisms into DKL, empowering the surrogates to adapt to contextual information gathered during the BO process. We combine this Transformer Deep Kernel with a learned acquisition function trained with continuous Soft Actor-Critic Reinforcement Learning to aid in exploration. This Reinforced Transformer Deep Kernel (RTDK-BO) approach yields state-of-the-art results in continuous high-dimensional optimization problems.