mlsanityData to 14 Oct 2026

Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models

arXiv 2305.16322 ↗First posted 25 May 2023cs.CV cs.GRcited by 359

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-Image diffusion models have made tremendous progress over the past two years, enabling the generation of highly realistic images based on open-domain text descriptions. However, despite their success, text descriptions often struggle to adequately convey detailed controls, even when composed of long and complex texts. Moreover, recent studies have also shown that these models face challenges in understanding such complex texts and generating the corresponding images. Therefore, there is a growing need to enable more control modes beyond text description. In this paper, we introduce Uni-ControlNet, a unified framework that allows for the simultaneous utilization of different local controls (e.g., edge maps, depth map, segmentation masks) and global controls (e.g., CLIP image embeddings) in a flexible and composable manner within one single model. Unlike existing methods, Uni-ControlNet only requires the fine-tuning of two additional adapters upon frozen pre-trained text-to-image diffusion models, eliminating the huge cost of training from scratch. Moreover, thanks to some dedicated adapter designs, Uni-ControlNet only necessitates a constant number (i.e., 2) of adapters, regardless of the number of local or global controls used. This not only reduces the fine-tuning costs and model size, making it more suitable for real-world deployment, but also facilitate composability of different conditions. Through both quantitative and qualitative comparisons, Uni-ControlNet demonstrates its superiority over existing methods in terms of controllability, generation quality and composability. Code is available at urlhttps://github.com/ShihaoZhaoZSH/Uni-ControlNet.

Key facts. Cited by 359 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 44 of its 58 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

44 of 58 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Cones: Concept Neurons in Diffusion Models for Customized Generation9 Mar
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
24 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  7. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  12. 2022Diffusion Models in Vision: A Surveycited by 1,015
  13. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  14. 2016Generative Adversarial Text to Image Synthesiscited by 735
  15. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  16. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  17. 2015Holistically-Nested Edge Detectioncited by 592
  18. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  19. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  20. 2021High-Fidelity Pluralistic Image Completion with Transformerscited by 134
  21. 2021M6: A Chinese Multimodal Pretrainercited by 91
  22. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesiscited by 75
  23. 2022Reduce Information Loss in Transformers for Pluralistic Image Inpaintingcited by 36
  24. 2021Towards Light-weight and Real-time Line Segment Detectioncited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 37; 2024: 124; 2025: 137; 2026: 61.

359 papers.

Sort:

By area:

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  2. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  3. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 436
  4. 2024OmniGen: Unified Image Generationcited by 394
  5. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  8. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  9. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  10. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  11. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 154
  12. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 143
38 more
  1. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  2. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 140
  3. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  4. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 120
  5. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
  6. 2024Transparent Image Layer Diffusion using Latent Transparencycited by 114
  7. 2024Improving Diffusion Models for Authentic Virtual Try-on in the Wildcited by 109
  8. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  9. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  10. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  11. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 91
  12. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  13. 2025DreamO: A Unified Framework for Image Customizationcited by 83
  14. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 75
  15. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  16. 2024DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representationscited by 68
  17. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 68
  18. 2023CoSeR: Bridging Image and Language for Cognitive Super-Resolutioncited by 66
  19. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 66
  20. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  21. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  22. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
  23. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  24. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  25. 2024CRS-Diff: Controllable Remote Sensing Image Generation with Diffusion Modelcited by 55
  26. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51
  27. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Modelscited by 49
  28. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 48
  29. 2024ControlAR: Controllable Image Generation with Autoregressive Modelscited by 48
  30. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  31. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  32. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 47
  33. 2023DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Controlcited by 45
  34. 2024Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Modelcited by 45
  35. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  36. 2025MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generationcited by 45
  37. 2025Reflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context Reflectioncited by 43
  38. 2023HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpaintingcited by 42

50 most cited of 359 All 359 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Localized Control in Diffusion Models via Latent Vector Prediction
  2. 2023Local Conditional Controlling for Text-to-Image Diffusion Models
  3. 2025Universal Few-Shot Spatial Control for Diffusion Models
  4. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  5. 2024ECNet: Effective Controllable Text-to-Image Diffusion Models
  6. 2023Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation
  7. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  8. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models