# End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control

arXiv: https://arxiv.org/abs/2308.01674 · HTML: https://mlsanity.com/paper/2308.01674/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 67 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [Adaptive Risk Minimization: Learning to Adapt to Domain Shift](https://mlsanity.com/paper/2007.02931/) (2020)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [Learning to Execute](https://mlsanity.com/paper/1410.4615/) (2014)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [Differentiating Through a Cone Program](https://mlsanity.com/paper/1904.09043/) (2019)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Generalizing Koopman Theory to allow for inputs and control](https://mlsanity.com/paper/1602.07647/) (2016)
- [Model Predictive Control of a Vehicle using Koopman Operator](https://mlsanity.com/paper/2103.04978/) (2021)
- [Koopman NMPC: Koopman-based Learning and Nonlinear Model Predictive Control of Control-affine Systems](https://mlsanity.com/paper/2105.08036/) (2021)
- [Data-driven End-to-end Learning of Pole Placement Control for Nonlinear Dynamics via Koopman Invariant Subspaces](https://mlsanity.com/paper/2208.08883/) (2022)

## Built on by (most cited first, 6 of 6)

- [Sample-Efficient Reinforcement Learning of Koopman eNMPC](https://mlsanity.com/paper/2503.18787/) (2025)
- [Task-optimal data-driven surrogate models for eNMPC via differentiable simulation and optimization](https://mlsanity.com/paper/2403.14425/) (2024)
- [Deep Neural Koopman Operator-based Economic Model Predictive Control of Shipboard Carbon Capture System](https://mlsanity.com/paper/2504.06818/) (2025)
- [Computationally efficient Gauss-Newton reinforcement learning for model predictive control](https://mlsanity.com/paper/2508.02441/) (2025)
- [Deep Koopman Economic Model Predictive Control of a Pasteurisation Unit](https://mlsanity.com/paper/2511.04437/) (2025)
- [End-to-End Reinforcement Learning of Koopman Models for eNMPC of an Air Separation Unit](https://mlsanity.com/paper/2511.04522/) (2025)

## Abstract (arXiv)

(Economic) nonlinear model predictive control ((e)NMPC) requires dynamic models that are sufficiently accurate and computationally tractable. Data-driven surrogate models for mechanistic models can reduce the computational burden of (e)NMPC; however, such models are typically trained by system identification for maximum prediction accuracy on simulation samples and perform suboptimally in (e)NMPC. We present a method for end-to-end reinforcement learning of Koopman surrogate models for optimal performance as part of (e)NMPC. We apply our method to two applications derived from an established nonlinear continuous stirred-tank reactor model. The controller performance is compared to that of (e)NMPCs utilizing models trained using system identification, and model-free neural network controllers trained using reinforcement learning. We show that the end-to-end trained models outperform those trained using system identification in (e)NMPC, and that, in contrast to the neural network controllers, the (e)NMPC controllers can react to changes in the control setting without retraining.