mlsanityData to 7 Oct 2026

ControlVideo: Training-free Controllable Text-to-Video Generation

arXiv 2305.13077 ↗First posted 22 May 2023cs.CVcited by 324

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-driven diffusion models have unlocked unprecedented abilities in image generation, whereas their video counterpart still lags behind due to the excessive training cost of temporal modeling. Besides the training burden, the generated videos also suffer from appearance inconsistency and structural flickers, especially in long video synthesis. To address these challenges, we design a training-free framework called ControlVideo to enable natural and efficient text-to-video generation. ControlVideo, adapted from ControlNet, leverages coarsely structural consistency from input motion sequences, and introduces three modules to improve video generation. Firstly, to ensure appearance coherence between frames, ControlVideo adds fully cross-frame interaction in self-attention modules. Secondly, to mitigate the flicker effect, it introduces an interleaved-frame smoother that employs frame interpolation on alternated frames. Finally, to produce long videos efficiently, it utilizes a hierarchical sampler that separately synthesizes each short clip with holistic coherency. Empowered with these modules, ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs quantitatively and qualitatively. Notably, thanks to the efficient designs, it generates both short and long videos within several minutes using one NVIDIA 2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.

Key facts. Cited by 324 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 40 of its 44 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

40 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  2. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Video Diffusion Models7 Apr

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
23 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  5. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  11. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  12. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  13. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  15. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  16. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  17. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  18. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  19. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  20. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  21. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  22. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 211
  23. 2022ImaginaryNet: Learning Object Detectors without Real Images and Annotationscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 49; 2024: 141; 2025: 96; 2026: 38.

324 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  3. 2025VACE: All-in-One Video Creation and Editingcited by 467
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  6. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  7. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  8. 2023A Survey on Video Diffusion Modelscited by 203
  9. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 202
  10. 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 202
  11. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  12. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
38 more
  1. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  2. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  3. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  4. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  5. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  6. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  7. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  8. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  9. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  10. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Qualitycited by 115
  11. 2024C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detectioncited by 111
  12. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  13. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  14. 2023Generative Image Dynamicscited by 93
  15. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  16. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  17. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  18. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  19. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  20. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  21. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  22. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  23. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  24. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 70
  25. 2025DiffuEraser: A Diffusion Model for Video Inpaintingcited by 69
  26. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  27. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
  28. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Modelscited by 64
  29. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 63
  30. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  31. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Modelscited by 58
  32. 2025Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizablecited by 58
  33. 2024Depth Any Video with Scalable Synthetic Datacited by 57
  34. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 56
  35. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  36. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  37. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  38. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51

50 most cited of 324 All 324 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning
  2. 2025Less is More: Data-Efficient Adaptation for Controllable Text-to-Video Generation
  3. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation
  4. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation
  5. 2023DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
  6. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  7. 2023AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
  8. 2025Controllable Video Generation: A Survey