# Non-linear State-space Model Identification from Video Data using Deep Encoders

arXiv: https://arxiv.org/abs/2012.07721 · HTML: https://mlsanity.com/paper/2012.07721/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 5 of 5)

- [Latent Neural ODEs with Sparse Bayesian Multiple Shooting](https://mlsanity.com/paper/2210.03466/) (2022)
- [Deep Subspace Encoders for Nonlinear System Identification](https://mlsanity.com/paper/2210.14816/) (2022)
- [Deep Identification of Nonlinear Systems in Koopman Form](https://mlsanity.com/paper/2110.02583/) (2021)
- [Initialization Approach for Nonlinear State-Space Identification via the Subspace Encoder Approach](https://mlsanity.com/paper/2304.02119/) (2023)
- [Learning Control-Affine Reduced-Order Models via Autoencoders](https://mlsanity.com/paper/2606.05045/) (2026)

## Abstract (arXiv)

Identifying systems with high-dimensional inputs and outputs, such as systems measured by video streams, is a challenging problem with numerous applications in robotics, autonomous vehicles and medical imaging. In this paper, we propose a novel non-linear state-space identification method starting from high-dimensional input and output data. Multiple computational and conceptual advances are combined to handle the high-dimensional nature of the data. An encoder function, represented by a neural network, is introduced to learn a reconstructability map to estimate the model states from past inputs and outputs. This encoder function is jointly learned with the dynamics. Furthermore, multiple computational improvements, such as an improved reformulation of multiple shooting and batch optimization, are proposed to keep the computational time under control when dealing with high-dimensional and large datasets. We apply the proposed method to a video stream of a simulated environment of a controllable ball in a unit box. The study shows low simulation error with excellent long term prediction capability of the model obtained using the proposed method.