mlsanityData to 14 Oct 2026

Koopman-Assisted Reinforcement Learning

arXiv 2403.02290 ↗First posted 4 Mar 2024cs.AI cs.LG math.DScited by 11

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The Bellman equation and its continuous form, the Hamilton-Jacobi-Bellman equation, are ubiquitous in reinforcement learning and control theory. However, these equations become intractable for high-dimensional or nonlinear systems. This paper develops two new reinforcement learning algorithms based on the data-driven Koopman operator, which lifts a nonlinear system into new coordinates where the dynamics become approximately linear, and where Hamilton-Jacobi-Bellman-based methods are more tractable. In particular, the Koopman operator captures the expectation of the time evolution of the value function via linear dynamics in the lifted coordinates. By parameterizing the Koopman operator with the control actions, we construct a ``controlled Koopman tensor'' that facilitates the estimation of the optimal value function. This enables us to reformulate two max-entropy RL algorithms: soft value iteration and soft actor-critic. This flexible and interpretable framework includes deterministic and stochastic systems, as well as discrete and continuous dynamics. Koopman Assisted reinforcement learning attains state-of-the-art performance with respect to traditional neural network-based soft actor-critic baselines on a linear state-space system, the Lorenz system, fluid flow past a cylinder, and a double-well potential with non-isotropic stochastic forcing.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 45 of its 106 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

45 of 106 references.

References · 45

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  9. 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 494
  10. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  12. 2021Modern Koopman Theory for Dynamical Systemscited by 276
33 more
  1. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  2. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  3. 2018A System for Massively Parallel Hyperparameter Tuningcited by 202
  4. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  5. 2017Learning Koopman Invariant Subspaces for Dynamic Mode Decompositioncited by 158
  6. 2017Linearly-Recurrent Autoencoder Networks for Learning Dynamicscited by 145
  7. 2016Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive controlcited by 140
  8. 2017Sparse identification of nonlinear dynamics for model predictive control in the low-data limitcited by 140
  9. 2017VAMPnets: Deep learning of molecular kineticscited by 136
  10. 2014Dynamic mode decomposition with controlcited by 129
  11. 2017Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systemscited by 127
  12. 2019Data-driven approximation of the Koopman generator: Model reduction, system identification, and controlcited by 85
  13. 2017On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operatorcited by 73
  14. 2018Efficient collective swimming by harnessing vortices through deep reinforcement learningcited by 73
  15. 2018Deep Dynamical Modeling and Control of Unsteady Fluid Flowscited by 68
  16. 2018Shared Autonomy via Deep Reinforcement Learningcited by 66
  17. 2017Data-driven discovery of Koopman eigenfunctions for controlcited by 64
  18. 2016Generalizing Koopman Theory to allow for inputs and controlcited by 53
  19. 2021Scientific multi-agent reinforcement learning for wall-models of turbulent flowscited by 52
  20. 2014Extracting spatial-temporal coherent patterns in large-scale neural recordings using dynamic mode decompositioncited by 49
  21. 2021Rigorous data-driven computation of spectral properties of Koopman operators for dynamical systemscited by 45
  22. 2022Residual Dynamic Mode Decomposition: Robust and verified Koopmanismcited by 36
  23. 2018Optimal construction of Koopman eigenfunctions for prediction and controlcited by 35
  24. 2017Flow Navigation by Smart Microswimmers via Reinforcement Learningcited by 34
  25. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  26. 2021Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamicscited by 27
  27. 2019Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learningcited by 21
  28. 2022The mpEDMD Algorithm for Data-Driven Computations of Measure-Preserving Dynamical Systemscited by 21
  29. 2016Variational Koopman models: slow collective variables and molecular kinetics from short off-equilibrium simulationscited by 18
  30. 2019Extended Dynamic Mode Decomposition with Learned Koopman Eigenfunctions for Prediction and Controlcited by 14
  31. 2023Beyond expectations: Residual Dynamic Mode Decomposition and Variance for Stochastic Dynamical Systemscited by 11
  32. 2018Data-driven Spatiotemporal Modal Decomposition for Time Frequency Analysiscited by 2
  33. 2019Online Learning of Dynamical Systems: An Operator Theoretic Approachcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 6.

11 papers.

  1. 2024Operator World Models for Reinforcement Learningcited by 7
  2. 2025Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systemscited by 7
  3. 2024Learning Noise-Robust Stable Koopman Operator for Control with Hankel DMDcited by 1
  4. 2025Koopman-Based Generalization of Deep Reinforcement Learning With Application to Wireless Communicationscited by 1
  5. 2026Koopman operator theory: fundamentals, control, and applicationscited by 1
  6. 2025SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learningcited by 0
  7. 2026Generalization Analysis and Method for Domain Generalization for a Family of Recurrent Neural Networkscited by 0
  8. 2026A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Studycited by 0
  9. 2026Koopman-based surrogate modeling for reinforcement-learning-control of Rayleigh-Benard convectioncited by 0
  10. 2026Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximizationcited by 0
  11. 2026SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator
  2. 2025Leveraging Reinforcement Learning and Koopman Theory for Enhanced Model Predictive Control Performance
  3. 2025KIPPO: Koopman-Inspired Proximal Policy Optimization
  4. 2024Model Predictive Control of Nonlinear Dynamics Using Online Adaptive Koopman Operators ↗
  5. 2026Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems
  6. 2026Interpreting Reinforcement Learning Model Behavior via Koopman with Control
  7. 2026LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilization
  8. 2026An Output Feedback Q-learning Algorithm for Optimal Control of Nonlinear Systems with Koopman Linear Embedding