Data to 7 Oct 2026

Decomposing Motion and Content for Natural Video Sequence Prediction

arXiv 1706.08033 ↗First posted 25 Jun 2017cs.CVcited by 168

Not on the map. Why?

Abstract (arXiv)

We propose a deep neural network for the prediction of future frames in natural video sequences. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos. Our model is built upon the Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modeling motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified motion features, which simplifies the task of prediction. Our model is end-to-end trainable over multiple time steps, and naturally learns to decompose motion and content without separate training. We evaluate the proposed network architecture on human activity videos using KTH, Weizmann action, and UCF-101 datasets. We show state-of-the-art performance in comparison to recent approaches. To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatiotemporal dynamics for pixel-level future prediction in natural videos.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 42; 2023: 33; 2024: 23; 2025: 10; 2026: 9.

168 papers.

Sort:

By area:

2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
2022Diffusion Probabilistic Modeling for Video Generationcited by 237
2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
2022SimVP: Simpler yet Better Video Predictioncited by 181
2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137

38 more

2021Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regressioncited by 112
2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
2022E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Contextcited by 76
2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
2021CCVS: Context-aware Controllable Video Synthesiscited by 68
2022Synthetic Data -- what, why and how?cited by 66
2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
2022When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learningcited by 64
2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
2021Predicting Video with VQVAEcited by 61
2022SinFusion: Training Diffusion Models on a Single Image or Videocited by 58
2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
2024On the Content Bias in Fr'echet Video Distancecited by 51
2021Stochastic Image-to-Video Synthesis using cINNscited by 49
2026World Action Models: The Next Frontier in Embodied AIcited by 46
2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
2021Controllable Animation of Fluid Elements in Still Imagescited by 44
2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
2023DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videoscited by 42
2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
2021Meta-Learning Dynamics Forecasting Using Task Inferencecited by 37
2022Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioningcited by 37
2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
2021Contrastively Disentangled Sequential Variational Autoencodercited by 34
2023SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTMcited by 32
2022Optimizing Video Prediction via Video Frame Interpolationcited by 28

50 most cited of 168 All 168 (JSON)