# Disentangling Propagation and Generation for Video Prediction

arXiv: https://arxiv.org/abs/1812.00452 · HTML: https://mlsanity.com/paper/1812.00452/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2018.

## Built on by (most cited first, 30 of 44)

- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [CCVS: Context-aware Controllable Video Synthesis](https://mlsanity.com/paper/2107.08037/) (2021)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis](https://mlsanity.com/paper/2107.02790/) (2021)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting](https://mlsanity.com/paper/2403.16536/) (2024)
- [EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task](https://mlsanity.com/paper/2104.10066/) (2021)
- [SLAMP: Stochastic Latent Appearance and Motion Prediction](https://mlsanity.com/paper/2108.02760/) (2021)
- [SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM](https://mlsanity.com/paper/2308.09891/) (2023)
- [Optimizing Video Prediction via Video Frame Interpolation](https://mlsanity.com/paper/2206.13454/) (2022)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [MMVP: Motion-Matrix-based Video Prediction](https://mlsanity.com/paper/2308.16154/) (2023)
- [Diverse Video Generation using a Gaussian Process Trigger](https://mlsanity.com/paper/2107.04619/) (2021)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2211.10412/) (2022)
- [HairFIT: Pose-Invariant Hairstyle Transfer via Flow-based Hair Alignment and Semantic-Region-Aware Inpainting](https://mlsanity.com/paper/2206.08585/) (2022)
- [Stochastic Video Prediction with Structure and Motion](https://mlsanity.com/paper/2203.10528/) (2022)
- [Panoptic Segmentation Forecasting](https://mlsanity.com/paper/2104.03962/) (2021)
- [Physics-informed generative neural network: an application to troposphere temperature prediction](https://mlsanity.com/paper/2107.06991/) (2021)
- [Self-Supervision by Prediction for Object Discovery in Videos](https://mlsanity.com/paper/2103.05669/) (2021)
- [Semantic Complete Scene Forecasting from a 4D Dynamic Point Cloud Sequence](https://mlsanity.com/paper/2312.08054/) (2023)
- [WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction](https://mlsanity.com/paper/2211.14308/) (2022)
- [Motion Graph Unleashed: A Novel Approach to Video Prediction](https://mlsanity.com/paper/2410.22288/) (2024)
- [Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion](https://mlsanity.com/paper/2101.10777/) (2021)
- [Joint Forecasting of Panoptic Segmentations with Difference Attention](https://mlsanity.com/paper/2204.07157/) (2022)
- [Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images](https://mlsanity.com/paper/2208.09463/) (2022)

## Abstract (arXiv)

A dynamic scene has two types of elements: those that move fluidly and can be predicted from previous frames, and those which are disoccluded (exposed) and cannot be extrapolated. Prior approaches to video prediction typically learn either to warp or to hallucinate future pixels, but not both. In this paper, we describe a computational model for high-fidelity video prediction which disentangles motion-specific propagation from motion-agnostic generation. We introduce a confidence-aware warping operator which gates the output of pixel predictions from a flow predictor for non-occluded regions and from a context encoder for occluded regions. Moreover, in contrast to prior works where confidence is jointly learned with flow and appearance using a single network, we compute confidence after a warping step, and employ a separate network to inpaint exposed regions. Empirical results on both synthetic and real datasets show that our disentangling approach provides better occlusion maps and produces both sharper and more realistic predictions compared to strong baselines.