# CKNet: A Convolutional Neural Network Based on Koopman Operator for Modeling Latent Dynamics from Pixels

arXiv: https://arxiv.org/abs/2102.10205 · HTML: https://mlsanity.com/paper/2102.10205/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 58 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Linearly-Recurrent Autoencoder Networks for Learning Dynamics](https://mlsanity.com/paper/1712.01378/) (2017)
- [Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control](https://mlsanity.com/paper/1611.03537/) (2016)
- [VAMPnets: Deep learning of molecular kinetics](https://mlsanity.com/paper/1710.06012/) (2017)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operator](https://mlsanity.com/paper/1703.04680/) (2017)
- [Deep Dynamical Modeling and Control of Unsteady Fluid Flows](https://mlsanity.com/paper/1805.07472/) (2018)
- [Deep Variational Koopman Models: Inferring Koopman Observations for Uncertainty-Aware Dynamics Modeling and Control](https://mlsanity.com/paper/1902.09742/) (2019)
- [Advantages of Bilinear Koopman Realizations for the Modeling and Control of Systems with Unknown Dynamics](https://mlsanity.com/paper/2010.09961/) (2020)
- [Graph Dynamical Networks for Unsupervised Learning of Atomic Scale Dynamics in Materials](https://mlsanity.com/paper/1902.06836/) (2019)
- [Deep Neural Networks with Koopman Operators for Modeling and Control of Autonomous Vehicles](https://mlsanity.com/paper/2007.02219/) (2020)
- [A kernel-based approach to molecular conformation analysis](https://mlsanity.com/paper/1809.11092/) (2018)
- [Analyzing Koopman approaches to physics-informed machine learning for long-term sea-surface temperature forecasting](https://mlsanity.com/paper/2010.00399/) (2020)
- [Applied Koopman Operator Theory for Power Systems Technology](https://mlsanity.com/paper/1706.00159/) (2017)
- [Reduced-Order Nonlinear Observers via Contraction Analysis and Convex Optimization](https://mlsanity.com/paper/2012.06158/) (2020)
- [Generic Properties of Koopman Eigenfunctions for Stable Fixed Points and Periodic Orbits](https://mlsanity.com/paper/2010.04008/) (2020)

## Built on by (most cited first, 5 of 5)

- [RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operator](https://mlsanity.com/paper/2409.03107/) (2024)
- [Self-Supervised Decomposition, Disentanglement and Prediction of Video Sequences while Interpreting Dynamics: A Koopman Perspective](https://mlsanity.com/paper/2110.00547/) (2021)
- [Deep Learning for Koopman Operator Estimation in Idealized Atmospheric Dynamics](https://mlsanity.com/paper/2409.06522/) (2024)
- [LINOCS: Lookahead Inference of Networked Operators for Continuous Stability](https://mlsanity.com/paper/2404.18267/) (2024)
- [Supervised DKRC with Images for Offline System Identification](https://mlsanity.com/paper/2109.02241/) (2021)

## Abstract (arXiv)

With the development of end-to-end control based on deep learning, it is important to study new system modeling techniques to realize dynamics modeling with high-dimensional inputs. In this paper, a novel Koopman-based deep convolutional network, called CKNet, is proposed to identify latent dynamics from raw pixels. CKNet learns an encoder and decoder to play the role of the Koopman eigenfunctions and modes, respectively. The Koopman eigenvalues can be approximated by eigenvalues of the learned state transition matrix. The deterministic convolutional Koopman network (DCKNet) and the variational convolutional Koopman network (VCKNet) are proposed to span some subspace for approximating the Koopman operator respectively. Because CKNet is trained under the constraints of the Koopman theory, the identified latent dynamics is in a linear form and has good interpretability. Besides, the state transition and control matrices are trained as trainable tensors so that the identified dynamics is also time-invariant. We also design an auxiliary weight term for reducing multi-step linearity and prediction losses. Experiments were conducted on two offline trained and four online trained nonlinear forced dynamical systems with continuous action spaces in Gym and Mujoco environment respectively, and the results show that identified dynamics are adequate for approximating the latent dynamics and generating clear images. Especially for offline trained cases, this work confirms CKNet from a novel perspective that we visualize the evolutionary processes of the latent states and the Koopman eigenfunctions with DCKNet and VCKNet separately to each task based on the same episode and results demonstrate that different approaches learn similar features in shapes.