# iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

arXiv: https://arxiv.org/abs/2107.02790 · HTML: https://mlsanity.com/paper/2107.02790/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 82 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 30 of 68)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)

## Built on by (most cited first, 30 of 42)

- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis](https://mlsanity.com/paper/2108.08827/) (2021)
- [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model](https://mlsanity.com/paper/2405.20222/) (2024)
- [PhysGen3D: Crafting a Miniature Interactive World from a Single Image](https://mlsanity.com/paper/2503.20746/) (2025)
- [Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos](https://mlsanity.com/paper/2407.16124/) (2024)
- [Controllable Animation of Fluid Elements in Still Images](https://mlsanity.com/paper/2112.03051/) (2021)
- [LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis](https://mlsanity.com/paper/2412.15214/) (2024)
- [Semi-Parametric Neural Image Synthesis](https://mlsanity.com/paper/2204.11824/) (2022)
- [DragAPart: Learning a Part-Level Motion Prior for Articulated Objects](https://mlsanity.com/paper/2403.15382/) (2024)
- [Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics](https://mlsanity.com/paper/2408.04631/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [Beyond the Frame: Generating 360 Panoramic Videos from Perspective Videos](https://mlsanity.com/paper/2504.07940/) (2025)
- [PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model](https://mlsanity.com/paper/2503.19913/) (2025)
- [SCVRL: Shuffled Contrastive Video Representation Learning](https://mlsanity.com/paper/2205.11710/) (2022)
- [Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images](https://mlsanity.com/paper/2303.08639/) (2023)
- [Diverse Generation from a Single Video Made Possible](https://mlsanity.com/paper/2109.08591/) (2021)
- [Automatic Animation of Hair Blowing in Still Portrait Photos](https://mlsanity.com/paper/2309.14207/) (2023)
- [Motion Modes: What Could Happen Next?](https://mlsanity.com/paper/2412.00148/) (2024)
- [DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships](https://mlsanity.com/paper/2410.10751/) (2024)
- [Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation](https://mlsanity.com/paper/2306.03988/) (2023)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)

## Abstract (arXiv)

How would a static scene react to a local poke? What are the effects on other parts of an object if you could locally push it? There will be distinctive movement, despite evident variations caused by the stochastic nature of our world. These outcomes are governed by the characteristic kinematics of objects that dictate their overall motion caused by a local interaction. Conversely, the movement of an object provides crucial information about its underlying distinctive kinematics and the interdependencies between its parts. This two-way relation motivates learning a bijective mapping between object kinematics and plausible future image sequences. Therefore, we propose iPOKE -- invertible Prediction of Object Kinematics -- that, conditioned on an initial frame and a local poke, allows to sample object kinematics and establishes a one-to-one correspondence to the corresponding plausible videos, thereby providing a controlled stochastic video synthesis. In contrast to previous works, we do not generate arbitrary realistic videos, but provide efficient control of movements, while still capturing the stochastic nature of our environment and the diversity of plausible outcomes it entails. Moreover, our approach can transfer kinematics onto novel object instances and is not confined to particular object classes. Our project page is available at https://bit.ly/3dJN4Lf.