# Learning to Do or Learning While Doing: Reinforcement Learning and Bayesian Optimisation for Online Continuous Tuning

arXiv: https://arxiv.org/abs/2306.03739 · HTML: https://mlsanity.com/paper/2306.03739/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 63 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Learning to Optimize Neural Nets](https://mlsanity.com/paper/1703.00441/) (2017)
- [Bayesian Optimization for Dynamic Problems](https://mlsanity.com/paper/1803.03432/) (2018)
- [Real-world challenges for multi-agent reinforcement learning in grid-interactive buildings](https://mlsanity.com/paper/2112.06127/) (2021)
- [Autonomous Control of a Particle Accelerator using Deep Reinforcement Learning](https://mlsanity.com/paper/2010.08141/) (2020)
- [Deep reinforcement learning for smart calibration of radio telescopes](https://mlsanity.com/paper/2102.03200/) (2021)
- [Online tuning and light source control using a physics-informed Gaussian process Adi](https://mlsanity.com/paper/1911.01538/) (2019)

## Built on by (most cited first, 2 of 2)

- [Cheetah: Bridging the Gap Between Machine Learning and Particle Accelerator Physics with High-Speed, Differentiable Simulations](https://mlsanity.com/paper/2401.05815/) (2024)
- [Action-Attentive Deep Reinforcement Learning for Autonomous Alignment of Beamlines](https://mlsanity.com/paper/2411.12183/) (2024)

## Abstract (arXiv)

Online tuning of real-world plants is a complex optimisation problem that continues to require manual intervention by experienced human operators. Autonomous tuning is a rapidly expanding field of research, where learning-based methods, such as Reinforcement Learning-trained Optimisation (RLO) and Bayesian optimisation (BO), hold great promise for achieving outstanding plant performance and reducing tuning times. Which algorithm to choose in different scenarios, however, remains an open question. Here we present a comparative study using a routine task in a real particle accelerator as an example, showing that RLO generally outperforms BO, but is not always the best choice. Based on the study's results, we provide a clear set of criteria to guide the choice of algorithm for a given tuning task. These can ease the adoption of learning-based autonomous tuning solutions to the operation of complex real-world plants, ultimately improving the availability and pushing the limits of operability of these facilities, thereby enabling scientific and engineering advancements.