# Spatiotemporal Observer Design for Predictive Learning of High-Dimensional Data

arXiv: https://arxiv.org/abs/2402.15284 · HTML: https://mlsanity.com/paper/2402.15284/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction](https://mlsanity.com/paper/1610.00081/) (2016)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model](https://mlsanity.com/paper/1706.03458/) (2017)
- [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://mlsanity.com/paper/1507.08750/) (2015)
- [Deep Learning for Spatio-Temporal Data Mining: A Survey](https://mlsanity.com/paper/1906.04928/) (2019)
- [Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data](https://mlsanity.com/paper/1612.08544/) (2016)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [A Review on Deep Learning Techniques for Video Prediction](https://mlsanity.com/paper/2004.05214/) (2020)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics](https://mlsanity.com/paper/1811.07490/) (2018)
- [MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions](https://mlsanity.com/paper/2103.02243/) (2021)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [From Pixels to Torques: Policy Learning with Deep Dynamical Models](https://mlsanity.com/paper/1502.02251/) (2015)
- [Folded Recurrent Neural Networks for Future Video Prediction](https://mlsanity.com/paper/1712.00311/) (2017)
- [Why ResNet Works? Residuals Generalize](https://mlsanity.com/paper/1904.01367/) (2019)
- [Generalization Bounds for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01487/) (2019)
- [Luenberger observers for discrete-time nonlinear systems](https://mlsanity.com/paper/2002.02674/) (2020)
- [MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.03010/) (2022)
- [A Survey on Spatial and Spatiotemporal Prediction Methods](https://mlsanity.com/paper/2012.13384/) (2020)
- [RAP-Net: Region Attention Predictive Network for Precipitation Nowcasting](https://mlsanity.com/paper/2110.01035/) (2021)

## Built on by (most cited first, 2 of 2)

- [Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting](https://mlsanity.com/paper/2602.05384/) (2026)
- [PhyxMamba: Chaotic System Reconstruction from Short Context Observations with Generative State-Space Models](https://mlsanity.com/paper/2505.23863/) (2025)

## Abstract (arXiv)

Although deep learning-based methods have shown great success in spatiotemporal predictive learning, the framework of those models is designed mainly by intuition. How to make spatiotemporal forecasting with theoretical guarantees is still a challenging issue. In this work, we tackle this problem by applying domain knowledge from the dynamical system to the framework design of deep learning models. An observer theory-guided deep learning architecture, called Spatiotemporal Observer, is designed for predictive learning of high dimensional data. The characteristics of the proposed framework are twofold: firstly, it provides the generalization error bound and convergence guarantee for spatiotemporal prediction; secondly, dynamical regularization is introduced to enable the model to learn system dynamics better during training. Further experimental results show that this framework could capture the spatiotemporal dynamics and make accurate predictions in both one-step-ahead and multi-step-ahead forecasting scenarios.