# Sample-Efficient Reinforcement Learning of Koopman eNMPC

arXiv: https://arxiv.org/abs/2503.18787 · HTML: https://mlsanity.com/paper/2503.18787/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [Model-Based Reinforcement Learning via Meta-Policy Optimization](https://mlsanity.com/paper/1809.05214/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks](https://mlsanity.com/paper/2107.00940/) (2021)
- [Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control](https://mlsanity.com/paper/2108.00128/) (2021)
- [Physics-Informed Model-Based Reinforcement Learning](https://mlsanity.com/paper/2212.02179/) (2022)
- [Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption](https://mlsanity.com/paper/2405.19014/) (2024)
- [On Rollouts in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.16918/) (2025)
- [Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control](https://mlsanity.com/paper/2108.11645/) (2021)
- [Modern Machine Learning Tools for Monitoring and Control of Industrial Processes: A Survey](https://mlsanity.com/paper/2209.11123/) (2022)
- [End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control](https://mlsanity.com/paper/2308.01674/) (2023)
- [Physics-Informed Neural Networks for Dynamic Process Operations with Limited Physical Knowledge and Data](https://mlsanity.com/paper/2406.01528/) (2024)
- [Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization](https://mlsanity.com/paper/2405.17983/) (2024)
- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [Task-optimal data-driven surrogate models for eNMPC via differentiable simulation and optimization](https://mlsanity.com/paper/2403.14425/) (2024)

## Built on by (most cited first, 2 of 2)

- [Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEs](https://mlsanity.com/paper/2604.06497/) (2026)
- [SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2607.11624/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) can be used to tune data-driven (economic) nonlinear model predictive controllers ((e)NMPCs) for optimal performance in a specific control task by optimizing the dynamic model or parameters in the policy's objective function or constraints, such as state bounds. However, the sample efficiency of RL is crucial, and to improve it, we combine a model-based RL algorithm with our published method that turns Koopman (e)NMPCs into automatically differentiable policies. We apply our approach to an eNMPC case study of a continuous stirred-tank reactor (CSTR) model from the literature. The approach outperforms benchmark methods, i.e., data-driven eNMPCs using models based on system identification without further RL tuning of the resulting policy, and neural network controllers trained with model-based RL, by achieving superior control performance and higher sample efficiency. Furthermore, utilizing partial prior knowledge about the system dynamics via physics-informed learning further increases sample efficiency.