# Task-Oriented Koopman-Based Control with Contrastive Encoder

arXiv: https://arxiv.org/abs/2309.16077 · HTML: https://mlsanity.com/paper/2309.16077/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions](https://mlsanity.com/paper/2109.06697/) (2021)
- [Data-driven discovery of Koopman eigenfunctions for control](https://mlsanity.com/paper/1707.01146/) (2017)
- [Online Optimal Control with Linear Dynamics and Predictions: Algorithms and Regret Analysis](https://mlsanity.com/paper/1906.11378/) (2019)
- [Contrasting Contrastive Self-Supervised Representation Learning Pipelines](https://mlsanity.com/paper/2103.14005/) (2021)
- [Optimal construction of Koopman eigenfunctions for prediction and control](https://mlsanity.com/paper/1810.08733/) (2018)
- [Deep Learning of Koopman Representation for Control](https://mlsanity.com/paper/2010.07546/) (2020)
- [Deep Koopman Operator with Control for Nonlinear Systems](https://mlsanity.com/paper/2202.08004/) (2022)
- [Model-Based Control Using Koopman Operators](https://mlsanity.com/paper/1709.01568/) (2017)
- [ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations](https://mlsanity.com/paper/2303.01092/) (2023)
- [DeepKoCo: Efficient latent planning with a task-relevant Koopman representation](https://mlsanity.com/paper/2011.12690/) (2020)
- [An Optimal Control Formulation of Pulse-Based Control Using Koopman Operator](https://mlsanity.com/paper/1707.08462/) (2017)
- [Data-Efficient Model Learning for Control with Jacobian-Regularized Dynamic-Mode Decomposition}](https://mlsanity.com/paper/2212.07885/) (2022)

## Built on by (most cited first, 8 of 8)

- [Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems](https://mlsanity.com/paper/2505.18671/) (2025)
- [Intelligent Sensing-to-Action for Robust Autonomy at the Edge: Opportunities and Challenges](https://mlsanity.com/paper/2502.02692/) (2025)
- [RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operator](https://mlsanity.com/paper/2409.03107/) (2024)
- [Koopman Dreamer: Spectrally Constrained Latent Dynamics for Stable World-Model Imagination](https://mlsanity.com/paper/2607.19719/) (2026)
- [Course Correcting Koopman Representations](https://mlsanity.com/paper/2310.15386/) (2023)
- [Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers](https://mlsanity.com/paper/2410.07584/) (2024)
- [Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world models](https://mlsanity.com/paper/2607.04464/) (2026)
- [SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning](https://mlsanity.com/paper/2502.15512/) (2025)

## Abstract (arXiv)

We present task-oriented Koopman-based control that utilizes end-to-end reinforcement learning and contrastive encoder to simultaneously learn the Koopman latent embedding, operator, and associated linear controller within an iterative loop. By prioritizing the task cost as the main objective for controller learning, we reduce the reliance of controller design on a well-identified model, which, for the first time to the best of our knowledge, extends Koopman control from low to high-dimensional, complex nonlinear systems, including pixel-based tasks and a real robot with lidar observations. Code and videos are available hrefhttps://sites.google.com/view/kpmlilatsupp/here.