mlsanityData to 7 Oct 2026

Learning Semantic-Aware Dynamics for Video Prediction

arXiv 2104.09762 ↗First posted 20 Apr 2021cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

We propose an architecture and training scheme to predict video frames by explicitly modeling dis-occlusions and capturing the evolution of semantically consistent regions in the video. The scene layout (semantic map) and motion (optical flow) are decomposed into layers, which are predicted and fused with their context to generate future layouts and motions. The appearance of the scene is warped from past frames using the predicted motion in co-visible regions; dis-occluded regions are synthesized with content-aware inpainting utilizing the predicted scene layout. The result is a predictive model that explicitly represents objects and learns their class-specific motion, which we evaluate on video prediction benchmarks.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

47 of 60 references.

References · 47

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  6. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  7. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  8. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  11. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  12. 2018Video-to-Video Synthesiscited by 388
35 more
  1. 2016Generating Videos with Scene Dynamicscited by 376
  2. 2016Dynamic Filter Networkscited by 318
  3. 2018Stochastic Video Generation with a Learned Priorcited by 244
  4. 2017Stochastic Variational Video Predictioncited by 234
  5. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209
  6. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  7. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  8. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  9. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  10. 2019Deep Flow-Guided Video Inpaintingcited by 116
  11. 2019Deep Video Inpaintingcited by 107
  12. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  13. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  14. 2019Unsupervised Moving Object Detection via Contextual Information Separationcited by 72
  15. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  16. 2018Pose Guided Human Video Generationcited by 67
  17. 2020Future Video Synthesis with Object Motion Predictioncited by 57
  18. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 54
  19. 2019Video Generation from Single Semantic Label Mapcited by 54
  20. 2020Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Predictioncited by 52
  21. 2017Predicting Deeper into the Future of Semantic Segmentationcited by 51
  22. 2017Deep Video Generation, Prediction and Completion of Human Action Sequencescited by 50
  23. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  24. 2018SDCNet: Video Prediction Using Spatially-Displaced Convolutioncited by 50
  25. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  26. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  27. 2017Unsupervised Learning of Long-Term Motion Dynamics for Videoscited by 37
  28. 2017Attentive Semantic Video Generation using Captionscited by 28
  29. 2018Learning to Forecast and Refine Residual Motion for Image-to-Video Generationcited by 27
  30. 2018Probabilistic Video Generation using Holistic Attribute Controlcited by 24
  31. 2020Learning to Manipulate Individual Objects in an Imagecited by 22
  32. 2017Predicting Scene Parsing and Motion Dynamics in the Futurecited by 20
  33. 2018Recurrent Flow-Guided Semantic Forecastingcited by 19
  34. 2018Bayesian Prediction of Future Street Scenes using Synthetic Likelihoodscited by 17
  35. 2018Future Semantic Segmentation with Convolutional LSTMcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 7; 2024: 5; 2025: 1; 2026: 1.

21 papers.

Sort:

By area:

  1. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  2. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  3. 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
  4. 2022Optimizing Video Prediction via Video Frame Interpolationcited by 28
  5. 2023MMVP: Motion-Matrix-based Video Predictioncited by 18
  6. 2023MOSO: Decomposing MOtion, Scene and Object for Video Predictioncited by 12
  7. 2023STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Predictioncited by 12
  8. 2022A unified model for continuous conditional video predictioncited by 7
  9. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  10. 2022WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Predictioncited by 5
  11. 2024Advection Augmented Convolutional Neural Networkscited by 5
  12. 2023Model-Based Reinforcement Learning with Isolated Imaginationscited by 4
9 more
  1. 2024Motion Graph Unleashed: A Novel Approach to Video Predictioncited by 4
  2. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  3. 2021Semantic Prediction: Which One Should Come First, Recognition or Prediction?cited by 1
  4. 2023Fast Fourier Inception Networks for Occluded Video Predictioncited by 1
  5. 2024Forecasting Future Videos from Novel Views via Disentangled 3D Scene Representationcited by 1
  6. 2022Video Extrapolation in Space and Timecited by 0
  7. 2022Motion and Context-Aware Audio-Visual Conditioned Video Predictioncited by 0
  8. 2025Flow and Depth Assisted Video Prediction with Latent Transformercited by 0
  9. 2026InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecastingcited by 0