# Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL

arXiv: https://arxiv.org/abs/2012.09737 · HTML: https://mlsanity.com/paper/2012.09737/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [Machine Learning in Nuclear Physics](https://mlsanity.com/paper/2112.02309/) (2021)
- [Robust Errant Beam Prognostics with Conditional Modeling for Particle Accelerators](https://mlsanity.com/paper/2312.10040/) (2023)
- [Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking](https://mlsanity.com/paper/2510.02490/) (2025)
- [Adaptive Machine Learning for Time-Varying Systems: Low Dimensional Latent Space Tuning](https://mlsanity.com/paper/2107.06207/) (2021)
- [Trend-Based SAC Beam Control Method with Zero-Shot in Superconducting Linear Accelerator](https://mlsanity.com/paper/2305.13869/) (2023)
- [Human-in-the-loop Reinforcement Learning for Data Quality Monitoring in Particle Physics Experiments](https://mlsanity.com/paper/2405.15508/) (2024)
- [Outlook Towards Deployable Continual Learning for Particle Accelerators](https://mlsanity.com/paper/2504.03793/) (2025)
- [Adaptive Model-Based Reinforcement Learning for Orbit Feedback Control in NSLS-II Storage Ring](https://mlsanity.com/paper/2601.03486/) (2026)

## Abstract (arXiv)

Reinforcement learning holds tremendous promise in accelerator controls. The primary goal of this paper is to show how this approach can be utilised on an operational level on accelerator physics problems. Despite the success of model-free reinforcement learning in several domains, sample-efficiency still is a bottle-neck, which might be encompassed by model-based methods. We compare well-suited purely model-based to model-free reinforcement learning applied to the intensity optimisation on the FERMI FEL system. We find that the model-based approach demonstrates higher representational power and sample-efficiency, while the asymptotic performance of the model-free method is slightly superior. The model-based algorithm is implemented in a DYNA-style using an uncertainty aware model, and the model-free algorithm is based on tailored deep Q-learning. In both cases, the algorithms were implemented in a way, which presents increased noise robustness as omnipresent in accelerator control problems. Code is released in https://github.com/MathPhysSim/FERMI_RL_Paper.