mlsanityData to 14 Oct 2026

Deep multi-scale video prediction beyond mean square error

arXiv 1511.05440 ↗First posted 17 Nov 2015cs.LG cs.CV stat.MLcited by 300

Not on the map. Why?

Abstract (arXiv)

Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics. This is why pixel-space video prediction may be viewed as a promising avenue for unsupervised feature learning. In addition, while optical flow has been a very studied problem in computer vision for a long time, future frame prediction is rarely approached. Still, many vision applications could benefit from the knowledge of the next frames of videos, that does not require the complexity of tracking every pixel trajectories. In this work, we train a convolutional network to generate future frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean Squared Error (MSE) loss function, we propose three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function. We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset

Key facts. Cited by 300 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 102; 2022: 58; 2023: 59; 2024: 34; 2025: 22; 2026: 25.

300 papers.

Sort:

By area:

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  4. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  5. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  6. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  7. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  8. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  9. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  10. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  11. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  12. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
38 more
  1. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  2. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 182
  3. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  4. 2025Video World Models with Long-term Spatial Memorycited by 136
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  7. 2021Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localizationcited by 113
  8. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  9. 2021Mean-Shifted Contrastive Loss for Anomaly Detectioncited by 88
  10. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  11. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 86
  12. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  13. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  14. 2023Siamese Masked Autoencoderscited by 76
  15. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  16. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  17. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  18. 2022Synthetic Data -- what, why and how?cited by 67
  19. 2021Playable Video Generationcited by 64
  20. 2021Self-supervised Video Transformercited by 64
  21. 2021Predicting Video with VQVAEcited by 61
  22. 2023Anomaly Detection with Conditioned Denoising Diffusion Modelscited by 56
  23. 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
  24. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 55
  25. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 55
  26. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  27. 2023Video Event Restoration Based on Keyframes for Video Anomaly Detectioncited by 54
  28. 2021Learning the Predictability of the Futurecited by 49
  29. 2024Advancing Video Anomaly Detection: A Concise Review and a New Datasetcited by 49
  30. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  31. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 42
  32. 2023GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videoscited by 40
  33. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 40
  34. 2022Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videoscited by 39
  35. 2021Meta-Learning Dynamics Forecasting Using Task Inferencecited by 38
  36. 2021Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwritingcited by 38
  37. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  38. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 35

50 most cited of 300 All 300 (JSON)