# Control-Informed Reinforcement Learning for Chemical Processes

arXiv: https://arxiv.org/abs/2408.13566 · HTML: https://mlsanity.com/paper/2408.13566/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 66 references that are in the map.
- First posted on arXiv 24 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Reinforcement Learning for Batch Bioprocess Optimization](https://mlsanity.com/paper/1904.07292/) (2019)
- [Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning](https://mlsanity.com/paper/2111.07171/) (2021)
- [CAQL: Continuous Action Q-Learning](https://mlsanity.com/paper/1909.12397/) (2019)
- [A Data-Driven Automatic Tuning Method for MPC under Uncertainty using Constrained Bayesian Optimization](https://mlsanity.com/paper/2011.11841/) (2020)
- [Chance Constrained Policy Optimization for Process Control and Optimization](https://mlsanity.com/paper/2008.00030/) (2020)
- [Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach](https://mlsanity.com/paper/2203.09661/) (2022)
- [An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems](https://mlsanity.com/paper/2307.11432/) (2023)
- [Machine learning for industrial sensing and control: A survey and practical perspective](https://mlsanity.com/paper/2401.13836/) (2024)
- [DiffLoop: Tuning PID controllers by differentiating through the feedback loop](https://mlsanity.com/paper/2106.10516/) (2021)
- [Stability-Preserving Automatic Tuning of PID Control with Reinforcement Learning](https://mlsanity.com/paper/2112.15187/) (2021)
- [Automated deep reinforcement learning for real-time scheduling strategy of multi-energy system integrated with post-carbon and direct-air carbon captured system](https://mlsanity.com/paper/2301.07768/) (2023)

## Built on by (most cited first, 9 of 9)

- [PC-Gym: Benchmark Environments For Process Control Problems](https://mlsanity.com/paper/2410.22093/) (2024)
- [Hierarchical RL-MPC for Demand Response Scheduling](https://mlsanity.com/paper/2502.13714/) (2025)
- [Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)](https://mlsanity.com/paper/2508.16474/) (2025)
- [Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering](https://mlsanity.com/paper/2510.24272/) (2025)
- [From Detection to Action: Using LLM Agents for Fault-Tolerant Control](https://mlsanity.com/paper/2606.28011/) (2026)
- [Soft Actor-Critic with Backstepping-Pretrained DeepONet for control of PDEs](https://mlsanity.com/paper/2507.04232/) (2025)
- [Grounding Large Language Models as Generalizable Policies in Network Control](https://mlsanity.com/paper/2512.11839/) (2025)
- [Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systems](https://mlsanity.com/paper/2512.18317/) (2025)
- [Atoms to Processes: The Role of Artificial Intelligence and Machine Learning in Chemical Engineering](https://mlsanity.com/paper/2610.02014/) (2026)

## Abstract (arXiv)

This work proposes a control-informed reinforcement learning (CIRL) framework that integrates proportional-integral-derivative (PID) control components into the architecture of deep reinforcement learning (RL) policies. The proposed approach augments deep RL agents with a PID controller layer, incorporating prior knowledge from control theory into the learning process. CIRL improves performance and robustness by combining the best of both worlds: the disturbance-rejection and setpoint-tracking capabilities of PID control and the nonlinear modeling capacity of deep RL. Simulation studies conducted on a continuously stirred tank reactor system demonstrate the improved performance of CIRL compared to both conventional model-free deep RL and static PID controllers. CIRL exhibits better setpoint-tracking ability, particularly when generalizing to trajectories outside the training distribution, suggesting enhanced generalization capabilities. Furthermore, the embedded prior control knowledge within the CIRL policy improves its robustness to unobserved system disturbances. The control-informed RL framework combines the strengths of classical control and reinforcement learning to develop sample-efficient and robust deep reinforcement learning algorithms, with potential applications in complex industrial systems.