# Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks

arXiv: https://arxiv.org/abs/1807.09245 · HTML: https://mlsanity.com/paper/1807.09245/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2018.

## Built on by (most cited first, 11 of 11)

- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [PhysGen3D: Crafting a Miniature Interactive World from a Single Image](https://mlsanity.com/paper/2503.20746/) (2025)
- [FT2TF: First-Person Statement Text-To-Talking Face Generation](https://mlsanity.com/paper/2312.05430/) (2023)
- [Dimensions of Motion: Monocular Prediction through Flow Subspaces](https://mlsanity.com/paper/2112.01502/) (2021)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [JNMR: Joint Non-linear Motion Regression for Video Frame Interpolation](https://mlsanity.com/paper/2206.04231/) (2022)
- [MultiModal Action Conditioned Video Generation](https://mlsanity.com/paper/2510.02287/) (2025)
- [Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis](https://mlsanity.com/paper/2102.13329/) (2021)

## Abstract (arXiv)

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods that have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, and on real-world video frames. We present analyses of the learned network representations, showing it is implicitly learning a compact encoding of object appearance and motion. We also demonstrate a few of its applications, including visual analogy-making and video extrapolation.