mlsanityData to 7 Oct 2026

FitVid: Overfitting in Pixel-Level Video Prediction

arXiv 2106.13195 ↗First posted 24 Jun 2021cs.CV cs.LGcited by 86

Not on the map. Why?

Abstract (arXiv)

An agent that is capable of predicting what happens next can perform a variety of tasks through planning with no additional training. Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for a variety of visual perception tasks. This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances. Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain. There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions. In this paper, we argue that the inefficient use of parameters in the current video models is the main reason for underfitting. Therefore, we introduce a new architecture, named FitVid, which is capable of severe overfitting on the common benchmarks while having similar parameter count as the current state-of-the-art models. We analyze the consequences of overfitting, illustrating how it can produce unexpected outcomes such as generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques. As a result, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 85 of its 115 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

85 of 115 references.

References · 85

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2016Density estimation using Real NVPcited by 2,029
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
38 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  3. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  4. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  5. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  11. 2018Video-to-Video Synthesiscited by 388
  12. 2016Generating Videos with Scene Dynamicscited by 376
  13. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  14. 2016Dynamic Filter Networkscited by 318
  15. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  16. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  17. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  18. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  19. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  20. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  21. 2018Stochastic Video Generation with a Learned Priorcited by 244
  22. 2017Stochastic Variational Video Predictioncited by 234
  23. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  24. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209
  25. 2018DeepFakes: a New Threat to Face Recognition? Assessment and Detectioncited by 208
  26. 2019Few-shot Video-to-Video Synthesiscited by 196
  27. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  28. 2019Scaling Autoregressive Video Modelscited by 189
  29. 2018Stochastic Adversarial Video Predictioncited by 173
  30. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  31. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  32. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  33. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  34. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  35. 2016Video Pixel Networkscited by 108
  36. 2020Stochastic Latent Residual Video Predictioncited by 106
  37. 2020Latent Video Transformercited by 106
  38. 2019Improved Conditional VRNNs for Video Predictioncited by 104

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 28; 2023: 23; 2024: 12; 2025: 15; 2026: 8.

86 papers.

Sort:

By area:

  1. 2022Video Diffusion Modelscited by 2,280
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  6. 2022MAGVIT: Masked Generative Video Transformercited by 341
  7. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  8. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  9. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  10. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  11. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  12. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
38 more
  1. 2023Video Language Planningcited by 160
  2. 2023Structured World Models from Human Videoscited by 144
  3. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  4. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  5. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  6. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  7. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  8. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  9. 2024Rolling Diffusion Modelscited by 102
  10. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  11. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  12. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
  13. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 53
  14. 2022Temporally Consistent Transformers for Video Generationcited by 43
  15. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  16. 2022Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioningcited by 37
  17. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  18. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34
  19. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 34
  20. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  21. 2023Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesiscited by 32
  22. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  23. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  24. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generationcited by 21
  25. 2022SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learningcited by 20
  26. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  27. 2023Multi-modal learning for geospatial vegetation forecastingcited by 16
  28. 2025World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertaintycited by 16
  29. 2023Video Diffusion Models with Local-Global Context Guidancecited by 15
  30. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  31. 2023Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenescited by 13
  32. 2023MOSO: Decomposing MOtion, Scene and Object for Video Predictioncited by 12
  33. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  34. 2023STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Predictioncited by 12
  35. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  36. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  37. 2023Multi-Object Manipulation via Object-Centric Neural Scattering Functionscited by 9
  38. 2024Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Promptscited by 9

50 most cited of 86 All 86 (JSON)