# Implicit Stacked Autoregressive Model for Video Prediction

arXiv: https://arxiv.org/abs/2303.07849 · HTML: https://mlsanity.com/paper/2303.07849/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction](https://mlsanity.com/paper/1610.00081/) (2016)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics](https://mlsanity.com/paper/1811.07490/) (2018)
- [Learning to Generate Long-term Future via Hierarchical Prediction](https://mlsanity.com/paper/1704.05831/) (2017)
- [Convolutional Tensor-Train LSTM for Spatio-temporal Learning](https://mlsanity.com/paper/2002.09131/) (2020)
- [MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions](https://mlsanity.com/paper/2103.02243/) (2021)
- [SLAMP: Stochastic Latent Appearance and Motion Prediction](https://mlsanity.com/paper/2108.02760/) (2021)
- [STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction](https://mlsanity.com/paper/2203.16084/) (2022)
- [FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANs](https://mlsanity.com/paper/1810.01325/) (2018)
- [Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction](https://mlsanity.com/paper/2104.06697/) (2021)
- [MIMO Is All You Need : A Strong Multi-In-Multi-Out Baseline for Video Prediction](https://mlsanity.com/paper/2212.04655/) (2022)
- [CrevNet: Conditionally Reversible Video Prediction](https://mlsanity.com/paper/1910.11577/) (2019)

## Built on by (most cited first, 14 of 14)

- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting](https://mlsanity.com/paper/2410.23159/) (2024)
- [Deterministic Guidance Diffusion Model for Probabilistic Weather Forecasting](https://mlsanity.com/paper/2312.02819/) (2023)
- [Advection Augmented Convolutional Neural Networks](https://mlsanity.com/paper/2406.19253/) (2024)
- [Everything is a Video: Unifying Modalities through Next-Frame Prediction](https://mlsanity.com/paper/2411.10503/) (2024)
- [USTEP: Spatio-Temporal Predictive Learning under A Unified View](https://mlsanity.com/paper/2310.05829/) (2023)
- [STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing](https://mlsanity.com/paper/2411.10198/) (2024)
- [A Multi-In-Single-Out Network for Video Frame Interpolation without Optical Flow](https://mlsanity.com/paper/2311.11602/) (2023)
- [SFANet: Spatial-Frequency Attention Network for Weather Forecasting](https://mlsanity.com/paper/2405.18849/) (2024)
- [CausalVE: Face Video Privacy Encryption via Causal Video Prediction](https://mlsanity.com/paper/2409.19306/) (2024)
- [Data-Driven Uncertainty-Aware Forecasting of Sea Ice Conditions in the Gulf of Ob Based on Satellite Radar Imagery](https://mlsanity.com/paper/2410.19782/) (2024)
- [Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers](https://mlsanity.com/paper/2510.20807/) (2025)
- [Physics-Constrained Neural Networks for Improved Short-Term Weather Forecasting: A Case Study over the South Pacific](https://mlsanity.com/paper/2606.17659/) (2026)

## Abstract (arXiv)

Future frame prediction has been approached through two primary methods: autoregressive and non-autoregressive. Autoregressive methods rely on the Markov assumption and can achieve high accuracy in the early stages of prediction when errors are not yet accumulated. However, their performance tends to decline as the number of time steps increases. In contrast, non-autoregressive methods can achieve relatively high performance but lack correlation between predictions for each time step. In this paper, we propose an Implicit Stacked Autoregressive Model for Video Prediction (IAM4VP), which is an implicit video prediction model that applies a stacked autoregressive method. Like non-autoregressive methods, stacked autoregressive methods use the same observed frame to estimate all future frames. However, they use their own predictions as input, similar to autoregressive methods. As the number of time steps increases, predictions are sequentially stacked in the queue. To evaluate the effectiveness of IAM4VP, we conducted experiments on three common future frame prediction benchmark datasets and weather&climate prediction benchmark datasets. The results demonstrate that our proposed model achieves state-of-the-art performance.