mlsanityData to 14 Oct 2026

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

arXiv 2112.02815 ↗First posted 6 Dec 2021cs.CV cs.CLcited by 88

Not on the map. Why?

Cited by 29 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Image-to-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA) structure to store appearance-motion aligned representation. To model the uncertainty and increase the diversity, it further allows the injection of explicit condition and implicit randomness. Through three-dimensional axial transformers, MA is interacted with given image to generate next frames recursively with satisfying controllability and diversity. Accompanying the new task, we build two new video-text paired datasets based on MNIST and CATER for evaluation. Experiments conducted on these datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task. Source code for model and datasets will be available soon.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

39 of 48 references.

References · 39

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  7. 2017The Kinetics Human Action Video Datasetcited by 1,909
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2016Deep Variational Information Bottleneckcited by 930
  10. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  11. 2018Video-to-Video Synthesiscited by 390
  12. 2016Generating Videos with Scene Dynamicscited by 376
27 more
  1. 2019Axial Attention in Multidimensional Transformerscited by 358
  2. 2018Everybody Dance Nowcited by 315
  3. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  4. 2017Stochastic Variational Video Predictioncited by 234
  5. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  6. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  7. 2019Few-shot Video-to-Video Synthesiscited by 197
  8. 2017Video Generation From Textcited by 154
  9. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  10. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  11. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 89
  12. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  13. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  14. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  15. 2018Pose Guided Human Video Generationcited by 67
  16. 2021Playable Video Generationcited by 64
  17. 2020Future Video Synthesis with Object Motion Predictioncited by 57
  18. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 55
  19. 2019Video Generation from Single Semantic Label Mapcited by 54
  20. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  21. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  22. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44
  23. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  24. 2017Attentive Semantic Video Generation using Captionscited by 28
  25. 2018Learning to Forecast and Refine Residual Motion for Image-to-Video Generationcited by 27
  26. 2019Vid2Game: Controllable Characters Extracted from Real-World Videoscited by 18
  27. 2021InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generationcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 16; 2024: 29; 2025: 26; 2026: 12.

88 papers.

Sort:

By area:

  1. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  3. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  4. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  5. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  6. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  7. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  8. 2024Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advancescited by 110
  9. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 104
  10. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  11. 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 52
  12. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 48
38 more
  1. 2023LivePhoto: Real Image Animation with Text-guided Motion Controlcited by 44
  2. 2024Autoregressive Models in Vision: A Surveycited by 39
  3. 2024StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videoscited by 37
  4. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 35
  5. 2023Towards Consistent Video Editing with Text-to-Image Diffusion Modelscited by 34
  6. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  7. 2024Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videoscited by 32
  8. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  9. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  10. 2024TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Modelscited by 25
  11. 2025Generative Physical AI in Vision: A Surveycited by 24
  12. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  13. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  14. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generationcited by 22
  15. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 21
  16. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19
  17. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  18. 2024UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Controlcited by 14
  19. 2024Dance Any Beat: Blending Beats with Visuals in Dance Video Generationcited by 14
  20. 2025DanceTogether! Identity-Preserving Multi-Person Interactive Video Generationcited by 13
  21. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  22. 2024VISAGE: Video Synthesis using Action Graphs for Surgerycited by 11
  23. 2024MotiF: Making Text Count in Image Animation with Motion Focal Losscited by 10
  24. 2025We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedbackcited by 10
  25. 2025Taming Camera-Controlled Video Generation with Verifiable Geometry Rewardcited by 10
  26. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 10
  27. 2024Kinetic Typography Diffusion Modelcited by 9
  28. 2024Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verificationcited by 9
  29. 2023Vision + Language Applications: A Surveycited by 8
  30. 2023Benchmarks for Physical Reasoning AIcited by 8
  31. 2024ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructionscited by 8
  32. 2025EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Posescited by 8
  33. 2023Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODEcited by 7
  34. 2023Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generationcited by 7
  35. 2024Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraintscited by 7
  36. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  37. 2022Controllable Video Generation through Global and Local Motion Dynamicscited by 6
  38. 2025C-Drag: Chain-of-Thought Driven Motion Controller for Video Generationcited by 6

50 most cited of 88 All 88 (JSON)