mlsanityData to 7 Oct 2026

ControlNeXt: Powerful and Efficient Control for Image and Video Generation

arXiv 2408.06070 ↗First posted 12 Aug 2024cs.CVcited by 158

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have demonstrated remarkable and robust abilities in both image and video generation. To achieve greater control over generated results, researchers introduce additional architectures, such as ControlNet, Adapters and ReferenceNet, to integrate conditioning controls. However, current controllable generation methods often require substantial additional computational resources, especially for video generation, and face challenges in training or exhibit weak control. In this paper, we propose ControlNeXt: a powerful and efficient method for controllable image and video generation. We first design a more straightforward and efficient architecture, replacing heavy additional branches with minimal additional cost compared to the base model. Such a concise structure also allows our method to seamlessly integrate with other LoRA weights, enabling style alteration without the need for additional training. As for training, we reduce up to 90% of learnable parameters compared to the alternatives. Furthermore, we propose another method called Cross Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and stable training convergence. We have conducted various experiments with different base models across images and videos, demonstrating the robustness of our method.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 64 of its 75 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

64 of 75 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback11 Apr
  2. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey7 Mar
  3. 2023MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion18 Nov
  4. 2023Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models1 Jun
  5. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild18 May
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  8. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2022Scalable Diffusion Models with Transformerscited by 5,931
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  8. 2023Consistency Modelscited by 1,778
  9. 2018Group Normalizationcited by 1,514
  10. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  13. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  14. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  15. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2022On Distillation of Guided Diffusion Modelscited by 721
  18. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  19. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  20. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  21. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 524
  22. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  23. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  24. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  25. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  26. 2024Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wildcited by 281
  27. 2020Component Divide-and-Conquer for Real-World Image Super-Resolutioncited by 243
  28. 2023SinSR: Diffusion-Based Image Super-Resolution in a Single Stepcited by 235
  29. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  30. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  31. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  32. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  33. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  34. 2013Stabilitycited by 119
  35. 2021MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolutioncited by 79
  36. 2017On weight initialization in deep neural networkscited by 67
  37. 2019CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routingcited by 29
  38. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 96; 2026: 31.

158 papers.

Sort:

By area:

  1. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  2. 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 202
  3. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  4. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  5. 2025Controllable Video Generation: A Surveycited by 93
  6. 2024Animate-X: Universal Character Image Animation with Enhanced Motion Representationcited by 86
  7. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
  8. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  9. 2024Depth Any Video with Scalable Synthetic Datacited by 57
  10. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 55
  11. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  12. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
38 more
  1. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generationcited by 41
  2. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  3. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  4. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  5. 2024DisPose: Disentangling Pose Guidance for Controllable Human Image Animationcited by 35
  6. 2025Efficient Diffusion Models: A Surveycited by 35
  7. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 33
  8. 2025MagicMirror: ID-Preserved Video Generation in Video Diffusion Transformerscited by 32
  9. 2024CAR: Controllable Autoregressive Modeling for Visual Generationcited by 29
  10. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  11. 2025RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wildcited by 28
  12. 2025FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarioscited by 25
  13. 2024Generative Inbetweening through Frame-wise Conditions-Driven Video Generationcited by 23
  14. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  15. 2024Group Diffusion Transformers are Unsupervised Multitask Learnerscited by 22
  16. 2025RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformerscited by 18
  17. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 17
  18. 2024DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Posescited by 16
  19. 2025SketchVideo: Sketch-based Video Generation and Editingcited by 15
  20. 2025Dual Prompting Image Restoration with Diffusion Transformerscited by 15
  21. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  22. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
  23. 2025FramePainter: Endowing Interactive Image Editing with Video Diffusion Priorscited by 13
  24. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  25. 2024MovieCharacter: A Tuning-Free Framework for Controllable Character Video Synthesiscited by 12
  26. 2025LayerAnimate: Layer-level Control for Animationcited by 12
  27. 2025EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusioncited by 12
  28. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  29. 2025DanceTogether! Identity-Preserving Multi-Person Interactive Video Generationcited by 12
  30. 2025StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animationcited by 12
  31. 2025VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learningcited by 12
  32. 2024Video Diffusion Transformers are In-Context Learnerscited by 11
  33. 2024CPA: Camera-pose-awareness Diffusion Transformer for Video Generationcited by 10
  34. 2024Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attentioncited by 10
  35. 2025DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Modelscited by 10
  36. 2025AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformerscited by 10
  37. 2025RealisMotion: Decomposed Human Motion Control and Video Generation in the World Spacecited by 10
  38. 2025Animate-X++: Universal Character Image Animation with Dynamic Backgroundscited by 10

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation
  2. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  3. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
  4. 2024CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
  5. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
  6. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models
  7. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  8. 2024Boosting Camera Motion Control for Video Diffusion Transformers