Self-Supervised Decomposition, Disentanglement and Prediction of Video Sequences while Interpreting Dynamics: A Koopman Perspective
Not on the map. Why?
Abstract (arXiv)
Human interpretation of the world encompasses the use of symbols to categorize sensory inputs and compose them in a hierarchical manner. One of the long-term objectives of Computer Vision and Artificial Intelligence is to endow machines with the capacity of structuring and interpreting the world as we do. Towards this goal, recent methods have successfully been able to decompose and disentangle video sequences into their composing objects and dynamics, in a self-supervised fashion. However, there has been a scarce effort in giving interpretation to the dynamics of the scene. We propose a method to decompose a video into moving objects and their attributes, and model each object's dynamics with linear system identification tools, by means of a Koopman embedding. This allows interpretation, manipulation and extrapolation of the dynamics of the different objects by employing the Koopman operator K. We test our method in various synthetic datasets and successfully forecast challenging trajectories while interpreting them.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 1 Oct 2021.
Builds on
27 of 34 references.
References · 27
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2020Object-Centric Learning with Slot Attentioncited by 865
- 2017Deep learning for universal linear embeddings of nonlinear dynamicscited by 491
- 2015Unsupervised Learning of Video Representations using LSTMscited by 434
- 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
- 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
- 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
- 2019Contrastive Learning of Structured World Modelscited by 206
- 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
- 2017Linearly-Recurrent Autoencoder Networks for Learning Dynamicscited by 142
15 more
- 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
- 2017VAMPnets: Deep learning of molecular kineticscited by 135
- 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
- 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
- 2020Forecasting Sequential Data using Consistent Koopman Autoencoderscited by 114
- 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
- 2018Deep Dynamical Modeling and Control of Unsteady Fluid Flowscited by 67
- 2019Learning Compositional Koopman Operators for Model-Based Controlcited by 67
- 2016Generalizing Koopman Theory to allow for inputs and controlcited by 49
- 2019Structured Object-Aware Physics Prediction for Video Modeling and Planningcited by 42
- 2018Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackerscited by 22
- 2019Deep Variational Koopman Models: Inferring Koopman Observations for Uncertainty-Aware Dynamics Modeling and Controlcited by 18
- 2019Graph Dynamical Networks for Unsupervised Learning of Atomic Scale Dynamics in Materialscited by 13
- 2020Learning Disentangled Representations of Video with Missing Datacited by 5
- 2021CKNet: A Convolutional Neural Network Based on Koopman Operator for Modeling Latent Dynamics from Pixelscited by 5
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.