# Cautious Bayesian Optimization for Efficient and Scalable Policy Search

arXiv: https://arxiv.org/abs/2011.09445 · HTML: https://mlsanity.com/paper/2011.09445/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [Local policy search with Bayesian optimization](https://mlsanity.com/paper/2106.11899/) (2021)
- [Geometry-aware Bayesian Optimization in Robotics using Riemannian Mat'ern Kernels](https://mlsanity.com/paper/2111.01460/) (2021)
- [On Safety in Safe Bayesian Optimization](https://mlsanity.com/paper/2403.12948/) (2024)
- [High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy](https://mlsanity.com/paper/2402.03104/) (2024)
- [Performance-driven Constrained Optimal Auto-Tuner for MPC](https://mlsanity.com/paper/2503.07127/) (2025)
- [Simulation-Aided Policy Tuning for Black-Box Robot Learning](https://mlsanity.com/paper/2411.14246/) (2024)
- [Automated Controller Calibration by Kalman Filtering](https://mlsanity.com/paper/2111.10832/) (2021)
- [Augmented Bayesian Policy Search](https://mlsanity.com/paper/2407.04864/) (2024)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Dependency Structure Search Bayesian Optimization for Decision Making Models](https://mlsanity.com/paper/2308.00629/) (2023)
- [Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents](https://mlsanity.com/paper/2507.13491/) (2025)

## Abstract (arXiv)

Sample efficiency is one of the key factors when applying policy search to real-world problems. In recent years, Bayesian Optimization (BO) has become prominent in the field of robotics due to its sample efficiency and little prior knowledge needed. However, one drawback of BO is its poor performance on high-dimensional search spaces as it focuses on global search. In the policy search setting, local optimization is typically sufficient as initial policies are often available, e.g., via meta-learning, kinesthetic demonstrations or sim-to-real approaches. In this paper, we propose to constrain the policy search space to a sublevel-set of the Bayesian surrogate model's predictive uncertainty. This simple yet effective way of constraining the policy update enables BO to scale to high-dimensional spaces (>100) as well as reduces the risk of damaging the system. We demonstrate the effectiveness of our approach on a wide range of problems, including a motor skills task, adapting deep RL agents to new reward signals and a sim-to-real task for an inverted pendulum system.