# Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

arXiv: https://arxiv.org/abs/1607.02586 · HTML: https://mlsanity.com/paper/1607.02586/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2016.

## Built on by (most cited first, 30 of 46)

- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Video Language Planning](https://mlsanity.com/paper/2310.10625/) (2023)
- [LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond](https://mlsanity.com/paper/2105.10422/) (2021)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021)
- [PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop](https://mlsanity.com/paper/2503.09595/) (2025)
- [Meta-Learning Dynamics Forecasting Using Task Inference](https://mlsanity.com/paper/2102.10271/) (2021)
- [Video Autoencoder: self-supervised disentanglement of static 3D structure and motion](https://mlsanity.com/paper/2110.02951/) (2021)
- [SLAMP: Stochastic Latent Appearance and Motion Prediction](https://mlsanity.com/paper/2108.02760/) (2021)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos](https://mlsanity.com/paper/2504.11732/) (2025)
- [Diverse Video Generation using a Gaussian Process Trigger](https://mlsanity.com/paper/2107.04619/) (2021)
- [Comparing Correspondences: Video Prediction with Correspondence-wise Losses](https://mlsanity.com/paper/2104.09498/) (2021)
- [Animation from Blur: Multi-modal Blur Decomposition with Motion Guidance](https://mlsanity.com/paper/2207.10123/) (2022)
- [Traffic4cast at NeurIPS 2021 -- Temporal and Spatial Few-Shot Transfer Learning in Gridded Geo-Spatial Processes](https://mlsanity.com/paper/2203.17070/) (2022)
- [Event-based Continuous Color Video Decompression from Single Frames](https://mlsanity.com/paper/2312.00113/) (2023)
- [Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images](https://mlsanity.com/paper/2303.08639/) (2023)
- [Learning Temporal Dynamics from Cycles in Narrated Video](https://mlsanity.com/paper/2101.02337/) (2021)
- [Sign Language Production with Latent Motion Transformer](https://mlsanity.com/paper/2312.12917/) (2023)
- [Unsupervised Co-part Segmentation through Assembly](https://mlsanity.com/paper/2106.05897/) (2021)
- [3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes](https://mlsanity.com/paper/2304.11470/) (2023)
- [MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Pose](https://mlsanity.com/paper/2210.07181/) (2022)
- [Learning Value Functions from Undirected State-only Experience](https://mlsanity.com/paper/2204.12458/) (2022)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [Self-Supervision by Prediction for Object Discovery in Videos](https://mlsanity.com/paper/2103.05669/) (2021)
- [Object-centric Video Prediction without Annotation](https://mlsanity.com/paper/2105.02799/) (2021)
- [Interpretable Deep Feature Propagation for Early Action Recognition](https://mlsanity.com/paper/2107.05122/) (2021)
- [S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Prediction](https://mlsanity.com/paper/2307.06701/) (2023)

## Abstract (arXiv)

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.