mlsanityData to 7 Oct 2026

Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model

arXiv 2404.09967 ↗First posted 15 Apr 2024cs.CV cs.AI cs.LGcited by 64

On the map: Diffusion models → Video generation

Abstract (arXiv)

ControlNets are widely used for adding spatial control to text-to-image diffusion models with different conditions, such as depth maps, scribbles/sketches, and human poses. However, when it comes to controllable video generation, ControlNets cannot be directly integrated into new backbones due to feature space mismatches, and training ControlNets for new backbones can be a significant burden for many users. Furthermore, applying ControlNets independently to different frames cannot effectively maintain object temporal consistency. To address these challenges, we introduce Ctrl-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffusion model through the adaptation of pretrained ControlNets. Ctrl-Adapter offers strong and diverse capabilities, including image and video control, sparse-frame video control, fine-grained patch-level multi-condition control (via an MoE router), zero-shot adaptation to unseen conditions, and supports a variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided motion control. With six diverse U-Net/DiT-based image/video diffusion models (SDXL, PixArt-α, I2VGen-XL, SVD, Latte, Hotshot-XL), Ctrl-Adapter matches the performance of pretrained ControlNets on COCO and achieves the state-of-the-art on DAVIS 2017 with significantly lower computation (< 10 GPU hours).

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 67 of its 91 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

67 of 91 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  2. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  3. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  4. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  6. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  7. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  8. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  10. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  11. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
4 more
  1. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  2. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  9. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  12. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  13. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  14. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  15. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  16. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  17. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  18. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  19. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  20. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  21. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  22. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  23. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  24. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  25. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  26. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  27. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  28. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  29. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  30. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  31. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  32. 2017Video Generation From Textcited by 153
  33. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  34. 2024PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Modelscited by 139
  35. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  36. 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 137
  37. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  38. 2018Learning to Forecast and Refine Residual Motion for Image-to-Video Generationcited by 27

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 35; 2026: 20.

64 papers.

Sort:

By area:

  1. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  2. 2025Controllable Video Generation: A Surveycited by 93
  3. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 73
  4. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 66
  5. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  6. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  7. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  8. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  9. 2025Efficient Diffusion Models: A Surveycited by 35
  10. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  11. 2024StableV2V: Stablizing Shape Consistency in Video-to-Video Editingcited by 25
  12. 2024CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generationcited by 21
38 more
  1. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  2. 2025PreciseCam: Precise Camera Control for Text-to-Image Generationcited by 13
  3. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  4. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  5. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  6. 2024Edicho: Consistent Image Editing in the Wildcited by 9
  7. 2025InterLCM: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restorationcited by 9
  8. 2025TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Visioncited by 9
  9. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  10. 2025BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignmentcited by 9
  11. 2024MVOC: a training-free multiple video object composition method with diffusion modelscited by 8
  12. 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 8
  13. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  14. 2025FlexTraj: Image-to-Video Generation with Flexible Point Trajectory Controlcited by 6
  15. 2026AVControl: Efficient Framework for Training Audio-Visual Controlscited by 5
  16. 2025On Equivariance and Fast Sampling in Video Diffusion Models Trained with Warped Noisecited by 4
  17. 2025Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individualscited by 4
  18. 2025Temporal In-Context Fine-Tuning with Temporal Reasoning for Versatile Control of Video Diffusion Modelscited by 4
  19. 2025PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolutioncited by 4
  20. 2026Diffusion-aided Extreme Video Compression with Lightweight Semantics Guidancecited by 4
  21. 2025What Are You Doing? A Closer Look at Controllable Human Video Generationcited by 3
  22. 2025EF-VI: Enhancing End-Frame Injection for Video Inbetweeningcited by 3
  23. 2025AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluationcited by 3
  24. 2025Uncertainty-Aware ControlNet: Bridging Domain Gaps with Synthetic Image Generationcited by 3
  25. 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 3
  26. 2026CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generationcited by 3
  27. 2024Adaptive Multi-Modal Control of Digital Human Hand Synthesis Using a Region-Aware Cycle Losscited by 2
  28. 2024Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Modelscited by 2
  29. 2025GeoMan: Temporally Consistent Human Geometry Estimation using Image-to-Video Diffusioncited by 2
  30. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 2
  31. 2025Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedbackcited by 1
  32. 2025UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolutioncited by 1
  33. 2025Glance: Accelerating Diffusion Models with 1 Samplecited by 1
  34. 2026IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animationcited by 1
  35. 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Modelscited by 1
  36. 2026AMShortcut: An Inference- and Training-Efficient Inverse Design Model for Amorphous Materialscited by 1
  37. 2026Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusioncited by 1
  38. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
  2. 2025Universal Few-Shot Spatial Control for Diffusion Models
  3. 2023LOVECon: Text-driven Training-Free Long Video Editing with ControlNet
  4. 2025Enabling Versatile Controls for Video Diffusion Models
  5. 2024EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation
  6. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis
  7. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generation
  8. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning