mlsanityData to 14 Oct 2026

Transition Constrained Bayesian Optimization via Markov Decision Processes

arXiv 2402.08406 ↗First posted 13 Feb 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Bayesian optimization is a methodology to optimize black-box functions. Traditionally, it focuses on the setting where you can arbitrarily query the search space. However, many real-life problems do not offer this flexibility; in particular, the search space of the next query may depend on previous ones. Example challenges arise in the physical sciences in the form of local movement constraints, required monotonicity in certain variables, and transitions influencing the accuracy of measurements. Altogether, such transition constraints necessitate a form of planning. This work extends classical Bayesian optimization via the framework of Markov Decision Processes. We iteratively solve a tractable linearization of our utility function using reinforcement learning to obtain a policy that plans ahead for the entire horizon. This is a parallel to the optimization of an acquisition function in policy space. The resulting policy is potentially history-dependent and non-Markovian. We showcase applications in chemical reactor optimization, informative path planning, machine calibration, and other synthetic examples.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 34 of its 84 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

34 of 84 references.

References · 34

  1. 2021Convex optimizationcited by 2,515
  2. 2018A Tutorial on Bayesian Optimizationcited by 744
  3. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  4. 2017Max-value Entropy Search for Efficient Bayesian Optimizationcited by 206
  5. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 204
  6. 2016Simple Bayesian Algorithms for Best Arm Identificationcited by 117
  7. 2014Best-Arm Identification in Linear Banditscited by 100
  8. 2019Sequential Experimental Design for Transductive Linear Banditscited by 91
  9. 2021Reward is enough for convex MDPscited by 84
  10. 2018Information Directed Sampling and Bandits with Heteroscedastic Noisecited by 76
  11. 2021Gaussian Process Bandit Optimization with Few Batchescited by 57
  12. 2022Joint Entropy Search for Maximally-Informed Bayesian Optimizationcited by 55
22 more
  1. 2022Joint Entropy Search for Multi-objective Bayesian Optimizationcited by 45
  2. 2020Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Treescited by 41
  3. 2016Truncated Variance Reduction: A Unified Approach to Bayesian Optimization and Level-Set Estimationcited by 40
  4. 2019BINOCULARS for Efficient, Nonmyopic Sequential Experimental Designcited by 35
  5. 2017Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimizationcited by 34
  6. 2021A Nonmyopic Approach to Cost-Constrained Bayesian Optimizationcited by 19
  7. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  8. 2021Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costscited by 18
  9. 2022Maximizing information from chemical engineering data sets: Applications to machine learningcited by 17
  10. 2023Random Exploration in Bayesian Optimization: Order-Optimal Regret and Computational Efficiencycited by 17
  11. 2022Combining Multi-Fidelity Modelling and Asynchronous Batch Bayesian Optimizationcited by 16
  12. 2022SnAKe: Bayesian Optimization with Pathwise Explorationcited by 14
  13. 2022Constraining Gaussian Processes to Systems of Linear Ordinary Differential Equationscited by 14
  14. 2022Experimental Design for Linear Functionals in Reproducing Kernel Hilbert Spacescited by 9
  15. 2023Learning Regions of Interest for Bayesian Optimization with Adaptive Level-Set Estimationcited by 8
  16. 2022Tuning Particle Accelerators with Safety Constraints using Bayesian Optimizationcited by 7
  17. 2022Movement Penalized Bayesian Optimization with Application to Wind Energy Systemscited by 7
  18. 2023MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learningcited by 6
  19. 2022Bayesian Optimisation for Active Monitoring of Air Pollutioncited by 5
  20. 2023Bayesian Optimisation Against Climate Change: Applications and Benchmarkscited by 3
  21. 2023Gaussian Processes for Monitoring Air-Quality in Kampalacited by 3
  22. 2023Practical Path-based Bayesian Optimizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024System-Aware Neural ODE Processes for Few-Shot Bayesian Optimizationcited by 3
  2. 2025The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learningcited by 3
  3. 2025Constrained multi-fidelity Bayesian optimization with automatic stop conditioncited by 2
  4. 2025A Guide to Bayesian Optimization in Bioprocess Engineeringcited by 1
  5. 2025BALLAST: Bayesian Active Learning with Look-ahead Amendment for Sea-drifter Trajectories under Spatio-Temporal Vector Fieldscited by 1
  6. 2025Efficient Personalization of Generative Models via Optimal Experimental Designcited by 0
  7. 2026Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning
  2. 2026Bayesian Optimization in Linear Time
  3. 2024Bayesian Optimization with Adaptive Kernels for Robot Control
  4. 2024Augmented Bayesian Policy Search
  5. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
  6. 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
  7. 2024Epsilon-Greedy Thompson Sampling to Bayesian Optimization
  8. 2023Bayesian Inverse Transition Learning for Offline Settings