mlsanityData to 7 Oct 2026

Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning

arXiv 1605.08104 ↗First posted 25 May 2016cs.LG cs.AI cs.CVcited by 203

Not on the map. Why?

Abstract (arXiv)

While great strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem of unsupervised learning - leveraging unlabeled examples to learn about the structure of a domain - remains a difficult unsolved challenge. Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world. We describe a predictive neural network ("PredNet") architecture that is inspired by the concept of "predictive coding" from the neuroscience literature. These networks learn to predict future frames in a video sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers. We show that these networks are able to robustly learn to predict the movement of synthetic (rendered) objects, and that in doing so, the networks learn internal representations that are useful for decoding latent object parameters (e.g. pose) that support object recognition with fewer training views. We also show that these networks can scale to complex natural image streams (car-mounted camera videos), capturing key aspects of both egocentric movement and the movement of objects in the visual scene, and the representation learned in this setting is useful for estimating the steering angle. Altogether, these results suggest that prediction represents a powerful framework for unsupervised learning, allowing for implicit learning of object and scene structure.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 49; 2022: 45; 2023: 41; 2024: 33; 2025: 20; 2026: 15.

203 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  4. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  6. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  7. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  8. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  9. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  10. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  11. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  12. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
38 more
  1. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  2. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  3. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  4. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 99
  5. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  6. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  7. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  8. 2023Siamese Masked Autoencoderscited by 75
  9. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 74
  10. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  11. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  12. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generationcited by 59
  13. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  14. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 53
  15. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  16. 2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
  17. 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
  18. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  19. 2021Associative Memories via Predictive Codingcited by 42
  20. 2021Clockwork Variational Autoencoderscited by 40
  21. 2023Neither hype nor gloom do DNNs justicecited by 40
  22. 2023GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videoscited by 39
  23. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  24. 2021EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction taskcited by 34
  25. 2022Learning on Arbitrary Graph Topologies via Predictive Codingcited by 33
  26. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  27. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  28. 2022Optimizing Video Prediction via Video Frame Interpolationcited by 28
  29. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  30. 2021Mitigating Memorization of Noisy Labels via Regularization between Representationscited by 26
  31. 2022Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learningcited by 26
  32. 2022Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learningcited by 26
  33. 2022Hybrid Predictive Coding: Inferring, Fast and Slowcited by 25
  34. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  35. 2021Revisiting Hierarchical Approach for Persistent Long-Term Video Predictioncited by 23
  36. 2022Unsupervised Visual Representation Learning by Synchronous Momentum Groupingcited by 22
  37. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  38. 2021Learning Semantic-Aware Dynamics for Video Predictioncited by 21

50 most cited of 203 All 203 (JSON)