# Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach

arXiv: https://arxiv.org/abs/2203.09661 · HTML: https://mlsanity.com/paper/2203.09661/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 58 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [A Survey of Deep Meta-Learning](https://mlsanity.com/paper/2010.03522/) (2020)
- [Reinforcement Learning for Batch Bioprocess Optimization](https://mlsanity.com/paper/1904.07292/) (2019)
- [Learning Memory-Based Control for Human-Scale Bipedal Locomotion](https://mlsanity.com/paper/2006.02402/) (2020)
- [Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning](https://mlsanity.com/paper/2111.07171/) (2021)
- [Deep Reinforcement Learning for Process Control: A Primer for Beginners](https://mlsanity.com/paper/2004.05490/) (2020)
- [A Meta-Reinforcement Learning Approach to Process Control](https://mlsanity.com/paper/2103.14060/) (2021)
- [DiffLoop: Tuning PID controllers by differentiating through the feedback loop](https://mlsanity.com/paper/2106.10516/) (2021)
- [Stability-Preserving Automatic Tuning of PID Control with Reinforcement Learning](https://mlsanity.com/paper/2112.15187/) (2021)

## Built on by (most cited first, 12 of 12)

- [Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delay](https://mlsanity.com/paper/2307.11436/) (2023)
- [PC-Gym: Benchmark Environments For Process Control Problems](https://mlsanity.com/paper/2410.22093/) (2024)
- [Control-Informed Reinforcement Learning for Chemical Processes](https://mlsanity.com/paper/2408.13566/) (2024)
- [Machine learning for industrial sensing and control: A survey and practical perspective](https://mlsanity.com/paper/2401.13836/) (2024)
- [Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering](https://mlsanity.com/paper/2510.24272/) (2025)
- [Meta-Reinforcement Learning for Adaptive Control of Second Order Systems](https://mlsanity.com/paper/2209.09301/) (2022)
- [ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries](https://mlsanity.com/paper/2401.05251/) (2024)
- [MAKO: Meta-Adaptive Koopman Operators for Learning-based Model Predictive Control of Parametrically Uncertain Nonlinear Systems](https://mlsanity.com/paper/2510.09042/) (2025)
- [Reinforcement Learning with Partial Parametric Model Knowledge](https://mlsanity.com/paper/2304.13223/) (2023)
- [Facilitating Reinforcement Learning for Process Control Using Transfer Learning: Overview and Perspectives](https://mlsanity.com/paper/2404.00247/) (2024)
- [Scenario-based Thermal Management Parametrization Through Deep Reinforcement Learning](https://mlsanity.com/paper/2408.02022/) (2024)
- [Model-Free Reinforcement Learning Control for Resilient Cyber-Physical Systems](https://mlsanity.com/paper/2606.19069/) (2026)

## Abstract (arXiv)

Meta-learning is a branch of machine learning which trains neural network models to synthesize a wide variety of data in order to rapidly solve new problems. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that can be used to tune proportional--integral controllers. Our meta-RL agent has a recurrent structure that accumulates "context" to learn a system's dynamics through a hidden state variable in closed-loop. This architecture enables the agent to automatically adapt to changes in the process dynamics. In tests reported here, the meta-RL agent was trained entirely offline on first order plus time delay systems, and produced excellent results on novel systems drawn from the same distribution of process dynamics used for training. A key design element is the ability to leverage model-based information offline during training in simulated environments while maintaining a model-free policy structure for interacting with novel processes where there is uncertainty regarding the true process dynamics. Meta-learning is a promising approach for constructing sample-efficient intelligent controllers.