# Towards practical reinforcement learning for tokamak magnetic control

arXiv: https://arxiv.org/abs/2307.11546 · HTML: https://mlsanity.com/paper/2307.11546/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 60 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Controlling Commercial Cooling Systems Using Reinforcement Learning](https://mlsanity.com/paper/2211.07357/) (2022)
- [Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment](https://mlsanity.com/paper/1908.02388/) (2019)
- [Optimizing Memory Mapping Using Deep Reinforcement Learning](https://mlsanity.com/paper/2305.07440/) (2023)

## Built on by (most cited first, 9 of 9)

- [Wasserstein Policy Optimization](https://mlsanity.com/paper/2505.00663/) (2025)
- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [XiHeFusion: Harnessing Large Language Models for Science Communication in Nuclear Fusion](https://mlsanity.com/paper/2502.05615/) (2025)
- [Offline Reinforcement Learning for Rotation Profile Control in Tokamaks](https://mlsanity.com/paper/2605.05857/) (2026)
- [Real-time virtual circuits for plasma shape control via neural network emulators](https://mlsanity.com/paper/2605.14939/) (2026)
- [Real-Time Applicability of Emulated Virtual Circuits for Tokamak Plasma Shape Control](https://mlsanity.com/paper/2509.01789/) (2025)
- [Real-time virtual circuits for plasma shape control via neural network emulators: integration and testing in the MAST-U PCS](https://mlsanity.com/paper/2608.26216/) (2026)
- [Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment](https://mlsanity.com/paper/2608.20834/) (2026)
- [Real-time virtual circuits for plasma shape control via neural network emulators: experimental demonstration on MAST Upgrade](https://mlsanity.com/paper/2608.28468/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has shown promising results for real-time control systems, including the domain of plasma magnetic control. However, there are still significant drawbacks compared to traditional feedback control approaches for magnetic confinement. In this work, we address key drawbacks of the RL method; achieving higher control accuracy for desired plasma properties, reducing the steady-state error, and decreasing the required time to learn new tasks. We build on top of citedegrave2022magnetic, and present algorithmic improvements to the agent architecture and training procedure. We present simulation results that show up to 65% improvement in shape accuracy, achieve substantial reduction in the long-term bias of the plasma current, and additionally reduce the training time required to learn new tasks by a factor of 3 or more. We present new experiments using the upgraded RL-based controllers on the TCV tokamak, which validate the simulation results achieved, and point the way towards routinely achieving accurate discharges using the RL approach.