mlsanityData to 14 Oct 2026

EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization

arXiv 2411.00171 ↗First posted 31 Oct 2024cs.LG math.OCcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

To avoid myopic behavior, multi-step lookahead Bayesian optimization (BO) algorithms consider the sequential nature of BO and have demonstrated promising results in recent years. However, owing to the curse of dimensionality, most of these methods make significant approximations or suffer scalability issues. This paper presents a novel reinforcement learning (RL)-based framework for multi-step lookahead BO in high-dimensional black-box optimization problems. The proposed method enhances the scalability and decision-making quality of multi-step lookahead BO by efficiently solving the sequential dynamic program of the BO process in a near-optimal manner using RL. We first introduce an Attention-DeepSets encoder to represent the state of knowledge to the RL agent and subsequently propose a multi-task, fine-tuning procedure based on end-to-end (encoder-RL) on-policy learning. We evaluate the proposed method, EARL-BO (Encoder Augmented RL for BO), on synthetic benchmark functions and hyperparameter tuning problems, finding significantly improved performance compared to existing multi-step lookahead and high-dimensional BO methods.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 49 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

28 of 49 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
  4. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 374
  5. 2023Unexpected Improvements to Expected Improvement for Bayesian Optimizationcited by 149
  6. 2019Convolutional Conditional Neural Processescited by 142
  7. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 141
  8. 2020Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retrainingcited by 123
  9. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  10. 2022Recent Advances in Bayesian Optimizationcited by 76
  11. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  12. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
16 more
  1. 2020Data-efficient Domain Randomization with Bayesian Optimizationcited by 47
  2. 2015GLASSES: Relieving The Myopia Of Bayesian Optimisationcited by 38
  3. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 32
  4. 2017Decentralized High-Dimensional Bayesian Optimization with Factor Graphscited by 23
  5. 2021Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimizationcited by 23
  6. 2020Efficient Rollout Strategies for Bayesian Optimizationcited by 19
  7. 2024Bayesian optimization as a flexible and efficient design framework for sustainable process systemscited by 15
  8. 2022SnAKe: Bayesian Optimization with Pathwise Explorationcited by 14
  9. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  10. 2021Kernel Identification Through Transformerscited by 13
  11. 2024Pretrained Optimization Model for Zero-Shot Black Box Optimizationcited by 13
  12. 2021Multi-Objective Constrained Optimization for Energy Applications via Tree Ensemblescited by 12
  13. 2023RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernelscited by 11
  14. 2023PG-LBO: Enhancing High-Dimensional Bayesian Optimization with Pseudo-Label and Gaussian Process Guidancecited by 9
  15. 2024Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximationcited by 8
  16. 2023On Estimating the Gradient of the Expected Information Gain in Bayesian Experimental Designcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineeringcited by 2
  2. 2025Direct Regret Optimization in Bayesian Optimizationcited by 0
  3. 2025BOOST: A Data-Driven Framework for the Automated Joint Selection of Kernel and Acquisition Functions in Bayesian Optimizationcited by 0
  4. 2026Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settingscited by 0
  5. 2026BayMOTH: Bayesian optiMizatiOn with meTa-lookahead -- a simple approacHcited by 0
  6. 2026How Many Initial Points Does Bayesian Optimization Need?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRL
  2. 2026Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batching
  3. 2025BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL
  4. 2024CAGES: Cost-Aware Gradient Entropy Search for Efficient Local Multi-Fidelity Bayesian Optimization
  5. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  6. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
  7. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  8. 2024Combining Automated Optimisation of Hyperparameters and Reward Shape