# Koopman Spectrum Nonlinear Regulators and Efficient Online Learning

arXiv: https://arxiv.org/abs/2106.15775 · HTML: https://mlsanity.com/paper/2106.15775/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 66 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Contextual Decision Processes with Low Bellman Rank are PAC-Learnable](https://mlsanity.com/paper/1610.09512/) (2016)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Learning Stable Deep Dynamics Models](https://mlsanity.com/paper/2001.06116/) (2020)
- [Information Theoretic Regret Bounds for Online Nonlinear Control](https://mlsanity.com/paper/2006.12466/) (2020)
- [Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning](https://mlsanity.com/paper/2006.08684/) (2020)
- [Active Learning for Nonlinear System Identification with Guarantees](https://mlsanity.com/paper/2006.10277/) (2020)
- [Towards Generalization and Simplicity in Continuous Control](https://mlsanity.com/paper/1703.02660/) (2017)
- [Learning Compositional Koopman Operators for Model-Based Control](https://mlsanity.com/paper/1910.08264/) (2019)
- [Data-driven discovery of Koopman eigenfunctions for control](https://mlsanity.com/paper/1707.01146/) (2017)
- [Operator-valued Kernels for Learning from Functional Response Data](https://mlsanity.com/paper/1510.08231/) (2015)
- [Optimal construction of Koopman eigenfunctions for prediction and control](https://mlsanity.com/paper/1810.08733/) (2018)
- [Active Learning of Dynamics for Data-Driven Control Using Koopman Operators](https://mlsanity.com/paper/1906.05194/) (2019)
- [Random Fourier Features for Operator-Valued Kernels](https://mlsanity.com/paper/1605.02536/) (2016)
- [Learning Dynamics Models with Stable Invariant Sets](https://mlsanity.com/paper/2006.08935/) (2020)
- [Learning Stable Models for Prediction and Control](https://mlsanity.com/paper/2005.04291/) (2020)
- [Kernel Analog Forecasting: Multiscale Test Problems](https://mlsanity.com/paper/2005.06623/) (2020)
- [Neural Dynamic Mode Decomposition for End-to-End Modeling of Nonlinear Dynamics](https://mlsanity.com/paper/2012.06191/) (2020)
- [Metric on Nonlinear Dynamical Systems with Perron-Frobenius Operators](https://mlsanity.com/paper/1805.12324/) (2018)
- [Lyceum: An efficient and scalable ecosystem for robot learning](https://mlsanity.com/paper/2001.07343/) (2020)

## Built on by (most cited first, 5 of 5)

- [Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics](https://mlsanity.com/paper/2111.01365/) (2021)
- [Data-driven End-to-end Learning of Pole Placement Control for Nonlinear Dynamics via Koopman Invariant Subspaces](https://mlsanity.com/paper/2208.08883/) (2022)
- [Dynamic Structure Estimation from Bandit Feedback using Nonvanishing Exponential Sums](https://mlsanity.com/paper/2206.00861/) (2022)
- [Modeling Nonlinear Dynamics in Continuous Time with Inductive Biases on Decay Rates and/or Frequencies](https://mlsanity.com/paper/2212.13033/) (2022)
- [Signatures Meet Dynamic Programming: Generalizing Bellman Equations for Trajectory Following](https://mlsanity.com/paper/2312.05547/) (2023)

## Abstract (arXiv)

Most modern reinforcement learning algorithms optimize a cumulative single-step cost along a trajectory. The optimized motions are often 'unnatural', representing, for example, behaviors with sudden accelerations that waste energy and lack predictability. In this work, we present a novel paradigm of controlling nonlinear systems via the minimization of the Koopman spectrum cost: a cost over the Koopman operator of the controlled dynamics. This induces a broader class of dynamical behaviors that evolve over stable manifolds such as nonlinear oscillators, closed loops, and smooth movements. We demonstrate that some dynamics characterizations that are not possible with a cumulative cost are feasible in this paradigm, which generalizes the classical eigenstructure and pole assignments to nonlinear decision making. Moreover, we present a sample efficient online learning algorithm for our problem that enjoys a sub-linear regret bound under some structural assumptions.