mlsanityData to 7 Oct 2026

Video (language) modeling: a baseline for generative models of natural videos

arXiv 1412.6604 ↗First posted 20 Dec 2014cs.LG cs.CVcited by 92

Not on the map. Why?

Abstract (arXiv)

We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequence, the model discovers both spatial and temporal correlations which are useful to represent complex deformations and motion patterns. The models we propose are largely borrowed from the language modeling literature, and adapted to the vision domain by quantizing the space of image patches into a large dictionary. We demonstrate the approach on both a filling and a generation task. For the first time, we show that, after training on natural videos, such a model can predict non-trivial motions over short video sequences.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 24; 2023: 16; 2024: 11; 2025: 9; 2026: 4.

92 papers.

Sort:

By area:

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  4. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  5. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  8. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  9. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  10. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  11. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  12. 2022SimVP: Simpler yet Better Video Predictioncited by 181
38 more
  1. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  2. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  3. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  4. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  5. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  6. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  7. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  8. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  9. 2021Predicting Video with VQVAEcited by 61
  10. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  11. 2024On the Content Bias in Fr'echet Video Distancecited by 51
  12. 2021Learning the Predictability of the Futurecited by 48
  13. 2023TEDi: Temporally-Entangled Diffusion for Long-Term Motion Synthesiscited by 48
  14. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  15. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 34
  16. 2022STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Predictioncited by 31
  17. 2024Scaling 4D Representationscited by 30
  18. 2021Revisiting Hierarchical Approach for Persistent Long-Term Video Predictioncited by 23
  19. 2021Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videoscited by 23
  20. 2021Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottleneckscited by 22
  21. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  22. 2025An Empirical Study of Autoregressive Pre-training from Videoscited by 21
  23. 2023Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Modelcited by 16
  24. 2023The Age of Synthetic Realities: Challenges and Opportunitiescited by 15
  25. 2025Fostering Video Reasoning via Next-Event Predictioncited by 15
  26. 2021Learning Temporal Dynamics from Cycles in Narrated Videocited by 13
  27. 2025Light-X: Generative 4D Video Rendering with Camera and Illumination Controlcited by 11
  28. 2021A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Predictioncited by 10
  29. 2022Continual Predictive Learning from Videoscited by 8
  30. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  31. 2021Self-Supervision by Prediction for Object Discovery in Videoscited by 6
  32. 2021CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learningcited by 6
  33. 2021Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structurecited by 6
  34. 2021Social Processes: Self-Supervised Meta-Learning over Conversational Groups for Forecasting Nonverbal Social Cuescited by 6
  35. 2021Review of Video Predictive Understanding: Early Action Recognition and Future Action Predictioncited by 5
  36. 2023S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Predictioncited by 5
  37. 2021ArrowGAN : Learning to Generate Videos by Learning Arrow of Timecited by 4
  38. 2021Deep Video Prediction for Time Series Forecastingcited by 4

50 most cited of 92 All 92 (JSON)