# SLAMP: Stochastic Latent Appearance and Motion Prediction

arXiv: https://arxiv.org/abs/2108.02760 · HTML: https://mlsanity.com/paper/2108.02760/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completion](https://mlsanity.com/paper/1811.04441/) (2018)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks](https://mlsanity.com/paper/1911.01655/) (2019)
- [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016)
- [Dual Motion GAN for Future-Flow Embedded Video Prediction](https://mlsanity.com/paper/1708.00284/) (2017)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)
- [Temporal Difference Variational Auto-Encoder](https://mlsanity.com/paper/1806.03107/) (2018)
- [Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks](https://mlsanity.com/paper/1607.02586/) (2016)
- [Disentangling Propagation and Generation for Video Prediction](https://mlsanity.com/paper/1812.00452/) (2018)

## Built on by (most cited first, 30 of 32)

- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images](https://mlsanity.com/paper/2207.11148/) (2022)
- [Efficient Video Prediction via Sparsely Conditioned Flow Matching](https://mlsanity.com/paper/2211.14575/) (2022)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [StretchBEV: Stretching Future Instance Prediction Spatially and Temporally](https://mlsanity.com/paper/2203.13641/) (2022)
- [Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning](https://mlsanity.com/paper/2203.02573/) (2022)
- [Video Diffusion Models with Local-Global Context Guidance](https://mlsanity.com/paper/2306.02562/) (2023)
- [Implicit Stacked Autoregressive Model for Video Prediction](https://mlsanity.com/paper/2303.07849/) (2023)
- [MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks](https://mlsanity.com/paper/2203.09303/) (2022)
- [Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation](https://mlsanity.com/paper/2501.15878/) (2025)
- [MOSO: Decomposing MOtion, Scene and Object for Video Prediction](https://mlsanity.com/paper/2303.03684/) (2023)
- [Improving Flow Matching by Aligning Flow Divergence](https://mlsanity.com/paper/2602.00869/) (2026)
- [Stochastic Video Prediction with Structure and Motion](https://mlsanity.com/paper/2203.10528/) (2022)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.03010/) (2022)
- [WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction](https://mlsanity.com/paper/2211.14308/) (2022)
- [MS-LSTM: Exploring Spatiotemporal Multiscale Representations in Video Prediction Domain](https://mlsanity.com/paper/2304.07724/) (2023)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [Disentangling Content and Motion for Text-Based Neural Video Manipulation](https://mlsanity.com/paper/2211.02980/) (2022)
- [TKN: Transformer-based Keypoint Prediction Network For Real-time Video Prediction](https://mlsanity.com/paper/2303.09807/) (2023)
- [Masked Autoregressive Model for Weather Forecasting](https://mlsanity.com/paper/2409.20117/) (2024)
- [Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction](https://mlsanity.com/paper/2412.04929/) (2024)
- [Efficient Continuous Video Flow Model for Video Prediction](https://mlsanity.com/paper/2412.05633/) (2024)
- [Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training](https://mlsanity.com/paper/2503.23185/) (2025)
- [Motion and Context-Aware Audio-Visual Conditioned Video Prediction](https://mlsanity.com/paper/2212.04679/) (2022)
- [State-space Decomposition Model for Video Prediction Considering Long-term Motion Trend](https://mlsanity.com/paper/2404.11576/) (2024)
- [Lightweight Stochastic Video Prediction via Hybrid Warping](https://mlsanity.com/paper/2412.03061/) (2024)
- [Resolving Spatio-Temporal Entanglement in Video Prediction via Multi-Modal Attention](https://mlsanity.com/paper/2501.16997/) (2025)

## Abstract (arXiv)

Motion is an important cue for video prediction and often utilized by separating video content into static and dynamic components. Most of the previous work utilizing motion is deterministic but there are stochastic methods that can model the inherent uncertainty of the future. Existing stochastic models either do not reason about motion explicitly or make limiting assumptions about the static part. In this paper, we reason about appearance and motion in the video stochastically by predicting the future based on the motion history. Explicit reasoning about motion without history already reaches the performance of current stochastic models. The motion history further improves the results by allowing to predict consistent dynamics several frames into the future. Our model performs comparably to the state-of-the-art models on the generic video prediction datasets, however, significantly outperforms them on two challenging real-world autonomous driving datasets with complex motion and dynamic background.