The Power of Next-Frame Prediction for Learning Physical Laws
Not on the map. Why?
Abstract (arXiv)
Next-frame prediction is a useful and powerful method for modelling and understanding the dynamics of video data. Inspired by the empirical success of causal language modelling and next-token prediction in language modelling, we explore the extent to which next-frame prediction serves as a strong foundational learning strategy (analogous to language modelling) for inducing an understanding of the visual world. In order to quantify the specific visual understanding induced by next-frame prediction, we introduce six diagnostic simulation video datasets derived from fundamental physical laws created by varying physical constants such as gravity and mass. We demonstrate that our models trained only on next-frame prediction are capable of predicting the value of these physical constants (e.g. gravity) without having been trained directly to learn these constants via a regression task. We find that the generative training phase alone induces a model state that can predict physical constants significantly better than that of a random model, improving the loss by a factor of between 1.28 to 6.24. We conclude that next-frame prediction shows great promise as a general learning strategy to induce understanding of the many `laws' that govern the visual domain without the need for explicit labelling.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 21 May 2024.
Builds on
29 of 40 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2017Neural Discrete Representation Learningcited by 4,712
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2017Deformable Convolutional Networkscited by 1,488
17 more
- 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
- 2019Large Scale Adversarial Representation Learningcited by 239
- 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
- 2017Deep Learning for Precipitation Nowcasting: A Benchmark and A New Modelcited by 182
- 2016Accelerating Eulerian Fluid Simulation With Convolutional Networkscited by 130
- 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
- 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
- 2020Learning Stable Deep Dynamics Modelscited by 111
- 2020Latent Video Transformercited by 106
- 2019Improved Conditional VRNNs for Video Predictioncited by 104
- 2017Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledgecited by 96
- 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
- 2021Meta-Learning Dynamics Forecasting Using Task Inferencecited by 38
- 2017Transformation-Based Models of Video Sequencescited by 16
- 2021Local Frequency Domain Transformer Networks for Video Predictioncited by 9
- 2019Frequency Domain Transformer Networks for Video Predictioncited by 6
- 2021DFPN: Deformable Frame Prediction Networkcited by 4
Built on by
Citing papers per year: 2025: 2.
2 papers.