# Make It Move: Controllable Image-to-Video Generation with Text Descriptions

arXiv: https://arxiv.org/abs/2112.02815 · HTML: https://mlsanity.com/paper/2112.02815/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 30 of 39)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Temporal Generative Adversarial Nets with Singular Value Clipping](https://mlsanity.com/paper/1611.06624/) (2016)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks](https://mlsanity.com/paper/1709.07592/) (2017)
- [The Pose Knows: Video Forecasting by Generating Pose Futures](https://mlsanity.com/paper/1705.00053/) (2017)
- [To Create What You Tell: Generating Videos from Captions](https://mlsanity.com/paper/1804.08264/) (2018)
- [Dual Motion GAN for Future-Flow Embedded Video Prediction](https://mlsanity.com/paper/1708.00284/) (2017)
- [Pose Guided Human Video Generation](https://mlsanity.com/paper/1807.11152/) (2018)
- [Playable Video Generation](https://mlsanity.com/paper/2101.12195/) (2021)
- [Future Video Synthesis with Object Motion Prediction](https://mlsanity.com/paper/2004.00542/) (2020)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)

## Built on by (most cited first, 30 of 88)

- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos](https://mlsanity.com/paper/2409.02095/) (2024)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [ID-Animator: Zero-Shot Identity-Preserving Human Video Generation](https://mlsanity.com/paper/2404.15275/) (2024)
- [Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances](https://mlsanity.com/paper/2410.18775/) (2024)
- [CelebV-Text: A Large-Scale Facial Text-Video Dataset](https://mlsanity.com/paper/2303.14717/) (2023)
- [AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance](https://mlsanity.com/paper/2311.12886/) (2023)
- [Seer: Language Instructed Video Prediction with Latent Diffusion Models](https://mlsanity.com/paper/2303.14897/) (2023)
- [Breathing Life Into Sketches Using Text-to-Video Priors](https://mlsanity.com/paper/2311.13608/) (2023)
- [LivePhoto: Real Image Animation with Text-guided Motion Control](https://mlsanity.com/paper/2312.02928/) (2023)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos](https://mlsanity.com/paper/2409.07447/) (2024)
- [Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency](https://mlsanity.com/paper/2503.20785/) (2025)
- [Towards Consistent Video Editing with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.17431/) (2023)
- [Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation](https://mlsanity.com/paper/2211.12824/) (2022)
- [Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos](https://mlsanity.com/paper/2404.17571/) (2024)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs](https://mlsanity.com/paper/2505.11842/) (2025)
- [TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2404.16306/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding](https://mlsanity.com/paper/2503.14935/) (2025)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [LaMD: Latent Motion Diffusion for Image-Conditional Video Generation](https://mlsanity.com/paper/2304.11603/) (2023)
- [AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction](https://mlsanity.com/paper/2406.06465/) (2024)
- [FrameBridge: Improving Image-to-Video Generation with Bridge Models](https://mlsanity.com/paper/2410.15371/) (2024)
- [ORV: 4D Occupancy-centric Robot Video Generation](https://mlsanity.com/paper/2506.03079/) (2025)
- [UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control](https://mlsanity.com/paper/2403.02332/) (2024)

## Abstract (arXiv)

Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Image-to-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA) structure to store appearance-motion aligned representation. To model the uncertainty and increase the diversity, it further allows the injection of explicit condition and implicit randomness. Through three-dimensional axial transformers, MA is interacted with given image to generate next frames recursively with satisfying controllability and diversity. Accompanying the new task, we build two new video-text paired datasets based on MNIST and CATER for evaluation. Experiments conducted on these datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task. Source code for model and datasets will be available soon.