# Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

arXiv: https://arxiv.org/abs/2111.07171 · HTML: https://mlsanity.com/paper/2111.07171/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 13 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Deep Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1511.04143/) (2015)
- [Reinforcement Learning for Batch Bioprocess Optimization](https://mlsanity.com/paper/1904.07292/) (2019)
- [Deep Reinforcement Learning for Process Control: A Primer for Beginners](https://mlsanity.com/paper/2004.05490/) (2020)
- [A Meta-Reinforcement Learning Approach to Process Control](https://mlsanity.com/paper/2103.14060/) (2021)
- [Optimal PID and Antiwindup Control Design as a Reinforcement Learning Problem](https://mlsanity.com/paper/2005.04539/) (2020)

## Built on by (most cited first, 24 of 24)

- [Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delay](https://mlsanity.com/paper/2307.11436/) (2023)
- [Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach](https://mlsanity.com/paper/2203.09661/) (2022)
- [An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems](https://mlsanity.com/paper/2307.11432/) (2023)
- [PC-Gym: Benchmark Environments For Process Control Problems](https://mlsanity.com/paper/2410.22093/) (2024)
- [Generative AI and Process Systems Engineering: The Next Frontier](https://mlsanity.com/paper/2402.10977/) (2024)
- [Control-Informed Reinforcement Learning for Chemical Processes](https://mlsanity.com/paper/2408.13566/) (2024)
- [Machine learning for industrial sensing and control: A survey and practical perspective](https://mlsanity.com/paper/2401.13836/) (2024)
- [Surrogate Empowered Sim2Real Transfer of Deep Reinforcement Learning for ORC Superheat Control](https://mlsanity.com/paper/2308.02765/) (2023)
- [Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior](https://mlsanity.com/paper/2310.14098/) (2023)
- [Hierarchical RL-MPC for Demand Response Scheduling](https://mlsanity.com/paper/2502.13714/) (2025)
- [Distributional Reinforcement Learning for Scheduling of Chemical Production Processes](https://mlsanity.com/paper/2203.00636/) (2022)
- [Combining Deep Reinforcement Learning with a Jerk-Bounded Trajectory Generator for Kinematically Constrained Motion Planning](https://mlsanity.com/paper/2410.20907/) (2024)
- [Deep Reinforcement Learning in Applied Control: Challenges, Analysis, and Insights](https://mlsanity.com/paper/2507.08196/) (2025)
- [Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)](https://mlsanity.com/paper/2508.16474/) (2025)
- [Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering](https://mlsanity.com/paper/2510.24272/) (2025)
- [Meta-Reinforcement Learning for Adaptive Control of Second Order Systems](https://mlsanity.com/paper/2209.09301/) (2022)
- [A modular framework for stabilizing deep reinforcement learning control](https://mlsanity.com/paper/2304.03422/) (2023)
- [Self-Tuning PID Control via a Hybrid Actor-Critic-Based Neural Structure for Quadcopter Control](https://mlsanity.com/paper/2307.01312/) (2023)
- [Facilitating Reinforcement Learning for Process Control Using Transfer Learning: Overview and Perspectives](https://mlsanity.com/paper/2404.00247/) (2024)
- [Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay](https://mlsanity.com/paper/2501.18201/) (2025)
- [Predictive reinforcement learning based adaptive PID controller](https://mlsanity.com/paper/2506.08509/) (2025)
- [Real Time Self-Tuning Adaptive Controllers on Temperature Control Loops using Event-based Game Theory](https://mlsanity.com/paper/2506.13164/) (2025)
- [Addressing Terminal Constraints in Data-Driven Demand Response Scheduling](https://mlsanity.com/paper/2605.14741/) (2026)
- [Atoms to Processes: The Role of Artificial Intelligence and Machine Learning in Chemical Engineering](https://mlsanity.com/paper/2610.02014/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) is an optimization-driven framework for producing control strategies for general dynamical systems without explicit reliance on process models. Good results have been reported in simulation. Here we demonstrate the challenges in implementing a state of the art deep RL algorithm on a real physical system. Aspects include the interplay between software and existing hardware; experiment design and sample efficiency; training subject to input constraints; and interpretability of the algorithm and control law. At the core of our approach is the use of a PID controller as the trainable RL policy. In addition to its simplicity, this approach has several appealing features: No additional hardware needs to be added to the control system, since a PID controller can easily be implemented through a standard programmable logic controller; the control law can easily be initialized in a "safe'' region of the parameter space; and the final product -- a well-tuned PID controller -- has a form that practitioners can reason about and deploy with confidence.