mlsanityData to 14 Oct 2026

SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models

arXiv 2311.16933 ↗First posted 28 Nov 2023cs.CVcited by 175

On the map: Diffusion models → Video generation

Abstract (arXiv)

The development of text-to-video (T2V), i.e., generating videos with a given text prompt, has been significantly advanced in recent years. However, relying solely on text prompts often results in ambiguous frame composition due to spatial uncertainty. The research community thus leverages the dense structure signals, e.g., per-frame depth/edge sequences, to enhance controllability, whose collection accordingly increases the burden of inference. In this work, we present SparseCtrl to enable flexible structure control with temporally sparse signals, requiring only one or a few inputs, as shown in Figure 1. It incorporates an additional condition encoder to process these sparse signals while leaving the pre-trained T2V model untouched. The proposed approach is compatible with various modalities, including sketches, depth maps, and RGB images, providing more practical control for video generation and promoting applications such as storyboarding, depth rendering, keyframe animation, and interpolation. Extensive experiments demonstrate the generalization of SparseCtrl on both original and personalized T2V generators. Codes and models will be publicly available at https://guoyww.github.io/projects/SparseCtrl .

Key facts. Cited by 175 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 60 of its 64 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

60 of 64 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  2. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  3. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  4. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  5. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  6. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  7. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation7 Sep
  8. 2023ModelScope Text-to-Video Technical Report12 Aug
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  10. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  11. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
7 more
  1. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  2. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  7. 2022Video Diffusion Models7 Apr

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
29 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  6. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  7. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  8. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  11. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  12. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  13. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  14. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  15. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  16. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 359
  17. 2022MAGVIT: Masked Generative Video Transformercited by 346
  18. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  19. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  20. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  21. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  22. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  23. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 225
  24. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  25. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  26. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  27. 2022CLIPascene: Scene Sketching with Different Types and Levels of Abstractioncited by 64
  28. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
  29. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesiscited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 73; 2025: 74; 2026: 25.

175 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  2. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  3. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  4. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  5. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  6. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  7. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  8. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  9. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 166
  10. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  11. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 140
  12. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
38 more
  1. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  2. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  3. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  4. 2025Long Context Tuning for Video Generationcited by 113
  5. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  6. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  7. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 98
  8. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 93
  9. 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attentioncited by 91
  10. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 74
  11. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  12. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  13. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  14. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 66
  15. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 64
  16. 2024Image Conductor: Precision Control for Interactive Video Synthesiscited by 63
  17. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 62
  18. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
  19. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  20. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  21. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 57
  22. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 56
  23. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  24. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  25. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  26. 2024CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibilitycited by 50
  27. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  28. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 48
  29. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 47
  30. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  31. 2024Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolationcited by 44
  32. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  33. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 42
  34. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  35. 2024TrackGo: A Flexible and Efficient Method for Controllable Video Generationcited by 39
  36. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Modelscited by 36
  37. 2024Generative Video Propagationcited by 36
  38. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 34

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning
  2. 2025Enabling Versatile Controls for Video Diffusion Models
  3. 2025Less is More: Data-Efficient Adaptation for Controllable Text-to-Video Generation
  4. 2023ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond
  5. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
  6. 2024Video Diffusion Transformers are In-Context Learners
  7. 2025Video Text Preservation with Synthetic Text-Rich Videos
  8. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion