mlsanityData to 7 Oct 2026

Adding Conditional Control to Text-to-Image Diffusion Models

arXiv 2302.05543 ↗First posted 10 Feb 2023cs.CV cs.AI cs.GRcited by 5,466

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.

Key facts. Cited by 5,466 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 84 of its 119 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

84 of 119 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  2. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  3. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  11. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  12. 2021Palette: Image-to-Image Diffusion Models10 Nov
1 more
  1. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  13. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  14. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  15. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  16. 2021Variational Diffusion Modelscited by 1,145
  17. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  18. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  19. 2016HyperNetworkscited by 1,003
  20. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  21. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  22. 2020Pre-Trained Image Processing Transformercited by 891
  23. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  24. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  25. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  26. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  27. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  28. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  29. 2015Holistically-Nested Edge Detectioncited by 590
  30. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  31. 2018Noise2Noise: Learning Image Restoration without Clean Datacited by 526
  32. 2023Zero-shot Image-to-Image Translationcited by 503
  33. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  34. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  35. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  36. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  37. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  38. 2017Toward Multimodal Image-to-Image Translationcited by 295

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 856; 2024: 1748; 2025: 1750; 2026: 1112.

5,466 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  4. 2024Depth Anything V2cited by 1,385
  5. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  10. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  11. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  12. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
38 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  3. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  9. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  10. 2023Segment Everything Everywhere All at Oncecited by 546
  11. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  12. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  13. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  14. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  15. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  16. 2025VACE: All-in-One Video Creation and Editingcited by 467
  17. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  18. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  19. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  20. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  21. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  22. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  23. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  24. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  25. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  26. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  27. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  28. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  29. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  30. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  31. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  32. 2024OmniGen: Unified Image Generationcited by 388
  33. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  34. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  35. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  36. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  37. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  38. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343

50 most cited of 5,466 All 5,466 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Universal Few-Shot Spatial Control for Diffusion Models
  2. 2024ECNet: Effective Controllable Text-to-Image Diffusion Models
  3. 2023Local Conditional Controlling for Text-to-Image Diffusion Models
  4. 2025Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks
  5. 2025NanoControl: A Lightweight Framework for Precise and Efficient Control in Diffusion Transformer
  6. 2026Towards Controllable Image Generation through Representation-Conditioned Diffusion Models
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
  8. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification