mlsanityData to 7 Oct 2026

Generating Videos with Scene Dynamics

arXiv 1609.02612 ↗First posted 8 Sep 2016cs.CV cs.GR cs.LGcited by 376

Not on the map. Why?

Cited by 70 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.

Key facts. Cited by 376 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 90; 2022: 83; 2023: 76; 2024: 63; 2025: 42; 2026: 20.

376 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  6. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  8. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  10. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  11. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  12. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
38 more
  1. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  2. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  3. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  4. 2022MAGVIT: Masked Generative Video Transformercited by 341
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  6. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  7. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  8. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  9. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  10. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  11. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  12. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  13. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  14. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  15. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  16. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  17. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  18. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  19. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  20. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  21. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  22. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  23. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  24. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  25. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  26. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  27. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  28. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  29. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  30. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  31. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  32. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  33. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  34. 2022Generating Long Videos of Dynamic Scenescited by 113
  35. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  36. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  37. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  38. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103

50 most cited of 376 All 376 (JSON)