# Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems

arXiv: https://arxiv.org/abs/2604.19980 · HTML: https://mlsanity.com/paper/2604.19980/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1708.06850/) (2017)
- [On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operator](https://mlsanity.com/paper/1703.04680/) (2017)
- [Deep Dynamical Modeling and Control of Unsteady Fluid Flows](https://mlsanity.com/paper/1805.07472/) (2018)
- [Generalizing Koopman Theory to allow for inputs and control](https://mlsanity.com/paper/1602.07647/) (2016)
- [Deep Learning of Koopman Representation for Control](https://mlsanity.com/paper/2010.07546/) (2020)
- [Online dynamic mode decomposition for time-varying systems](https://mlsanity.com/paper/1707.02876/) (2017)
- [Linear identification of nonlinear systems: A lifting technique based on the Koopman operator](https://mlsanity.com/paper/1605.04457/) (2016)
- [Deep Koopman Learning of Nonlinear Time-Varying Systems](https://mlsanity.com/paper/2210.06272/) (2022)
- [Model Predictive Inferential Control of Neural State-Space Models for Autonomous Vehicle Motion Planning](https://mlsanity.com/paper/2310.08045/) (2023)
- [Koopman global linearization of contact dynamics for robot locomotion and manipulation enables elaborate control](https://mlsanity.com/paper/2511.06515/) (2025)

## Abstract (arXiv)

This paper presents a model-based reinforcement learning (RL) framework for optimal closed-loop control of nonlinear robotic systems. The proposed approach learns linear lifted dynamics through Koopman operator theory and integrates the resulting model into an actor-critic architecture for policy optimization, where the policy represents a parameterized closed-loop controller. To reduce computational cost and mitigate model rollout errors, policy gradients are estimated using one-step predictions of the learned dynamics rather than multi-step propagation. This leads to an online mini-batch policy gradient framework that enables policy improvement from streamed interaction data. The proposed framework is evaluated on several simulated nonlinear control benchmarks and two real-world hardware platforms, including a Kinova Gen3 robotic arm and a Unitree Go1 quadruped. Experimental results demonstrate improved sample efficiency over model-free RL baselines, superior control performance relative to model-based RL baselines, and control performance comparable to classical model-based methods that rely on exact system dynamics.