# Optimal PID and Antiwindup Control Design as a Reinforcement Learning Problem

arXiv: https://arxiv.org/abs/2005.04539 · HTML: https://mlsanity.com/paper/2005.04539/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2020.

## Built on by (most cited first, 4 of 4)

- [Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning](https://mlsanity.com/paper/2111.07171/) (2021)
- [DiffLoop: Tuning PID controllers by differentiating through the feedback loop](https://mlsanity.com/paper/2106.10516/) (2021)
- [Robust nonlinear set-point control with reinforcement learning](https://mlsanity.com/paper/2304.10277/) (2023)
- [Improving a Proportional Integral Controller with Reinforcement Learning on a Throttle Valve Benchmark](https://mlsanity.com/paper/2402.13654/) (2024)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has seen several successful applications to process control. Common methods rely on a deep neural network structure to model the controller or process. With increasingly complicated control structures, the closed-loop stability of such methods becomes less clear. In this work, we focus on the interpretability of DRL control methods. In particular, we view linear fixed-structure controllers as shallow neural networks embedded in the actor-critic framework. PID controllers guide our development due to their simplicity and acceptance in industrial practice. We then consider input saturation, leading to a simple nonlinear control structure. In order to effectively operate within the actuator limits we then incorporate a tuning parameter for anti-windup compensation. Finally, the simplicity of the controller allows for straightforward initialization. This makes our method inherently stabilizing, both during and after training, and amenable to known operational PID gains.