mlsanityData to 7 Oct 2026

Controllable Generation with Text-to-Image Diffusion Models: A Survey

arXiv 2403.04279 ↗First posted 7 Mar 2024cs.CVcited by 82

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 247 of its 372 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

247 of 372 references.

Closest ancestors · 90

Same direction, newest first

  1. 2024DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models27 Feb
  2. 2024InstanceDiffusion: Instance-level Control for Image Generation5 Feb
  3. 2024Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive16 Jan
  4. 2024Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation2 Jan
  5. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation21 Dec
  6. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model19 Dec
  7. 2023SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing18 Dec
  8. 2023Local Conditional Controlling for Text-to-Image Diffusion Models14 Dec
  9. 2023Compositional Inversion for Stable Diffusion Models13 Dec
  10. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition12 Dec
  11. 2023CCM: Adding Conditional Controls to Text-to-Image Consistency Models12 Dec
  12. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization11 Dec
38 more
  1. 2023UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models8 Dec
  2. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervision6 Dec
  3. 2023LooseControl: Lifting ControlNet for Generalized Depth Conditioning5 Dec
  4. 2023Orthogonal Adaptation for Modular Customization of Diffusion Models5 Dec
  5. 2023X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model4 Dec
  6. 2023Improving In-Context Learning in Diffusion Models with Visual Context-Modulated Prompts3 Dec
  7. 2023Meta ControlNet: Enhancing Task Adaptation via Meta Learning3 Dec
  8. 2023HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models30 Nov
  9. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation28 Nov
  10. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering28 Nov
  11. 2023CatVersion: Concatenating Embeddings for Diffusion-Based Text-to-Image Personalization24 Nov
  12. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Model24 Nov
  13. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis21 Nov
  14. 2023An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesis20 Nov
  15. 2023High-fidelity Person-centric Subject-to-Image Synthesis17 Nov
  16. 2023AnyText: Multilingual Visual Text Generation And Editing6 Nov
  17. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment17 Oct
  18. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation13 Oct
  19. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation26 Sep
  20. 2023MagiCapture: High-Resolution Multi-Concept Portrait Customization13 Sep
  21. 2023PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models11 Sep
  22. 2023Create Your World: Lifelong Text-to-Image Diffusion8 Sep
  23. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  24. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation20 Aug
  25. 2023ControlCom: Controllable Image Composition using Diffusion Model19 Aug
  26. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  27. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning21 Jul
  28. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursing20 Jul
  29. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  30. 2023Generate Anything Anywhere in Any Scene29 Jun
  31. 2023Text-Anchored Score Composition: Tackling Condition Misalignment in Text-to-Image Diffusion Models26 Jun
  32. 2023Zero-shot spatial layout conditioning for text-to-image diffusion models23 Jun
  33. 2023Continuous Layout Editing of Single Images with Diffusion Models22 Jun
  34. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment15 Jun
  35. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuning12 Jun
  36. 2023Grounded Text-to-Image Synthesis with Attention Refocusing8 Jun
  37. 2023Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation1 Jun
  38. 2023Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation1 Jun

50 newest of 90 All 90 (JSON)

Other references · 157

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2020Denoising Diffusion Implicit Modelscited by 9,655
  12. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
38 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2023Segment Anythingcited by 8,158
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  13. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  14. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  15. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  16. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  17. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  18. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  19. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  20. 2016Improved Techniques for Training GANscited by 2,938
  21. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  22. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  23. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  24. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  25. 2019Searching for MobileNetV3cited by 1,984
  26. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  27. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  28. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  29. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  30. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  31. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  32. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  33. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  34. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  35. 2022Diffusion Models in Vision: A Surveycited by 1,008
  36. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  37. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  38. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895

50 most cited of 157 All 157 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 30; 2026: 29.

82 papers.

Sort:

By area:

  1. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  2. 2024MV-Adapter: Multi-view Consistent Image Generation Made Easycited by 111
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  4. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  5. 2024Any-Resolution AI-Generated Image Detection by Spectral Learningcited by 56
  6. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  7. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  9. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  10. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  11. 20253D Scene Generation: A Surveycited by 24
  12. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
38 more
  1. 2024Where's Waldo: Diffusion Features for Personalized Segmentation and Retrievalcited by 18
  2. 2025D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samplescited by 14
  3. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmarkcited by 13
  4. 2025BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generationcited by 13
  5. 2025Concept Corrector: Erase concepts on the fly for text-to-image diffusion modelscited by 11
  6. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  7. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  8. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  9. 2025PromptMap: An Alternative Interaction Style for AI-Based Image Generationcited by 8
  10. 2025Image Editing with Diffusion Models: A Surveycited by 8
  11. 2024Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Modelscited by 7
  12. 2025PixelPonder: Dynamic Patch Adaptation for Enhanced Multi-Conditional Text-to-Image Generationcited by 7
  13. 2024AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customizationcited by 6
  14. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  15. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
  16. 2025Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approachcited by 5
  17. 2025Context-Aware Autoregressive Models for Multi-Conditional Image Generationcited by 5
  18. 2026PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generationcited by 5
  19. 2026HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Imagescited by 5
  20. 2026COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflowscited by 5
  21. 2024Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspectivecited by 4
  22. 2025A Comprehensive Survey on Visual Concept Mining in Text-to-image Diffusion Modelscited by 4
  23. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  24. 2024OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentationcited by 3
  25. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motioncited by 3
  26. 2025Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Modelscited by 3
  27. 2025Exploring Position Encoding in Diffusion U-Net for Training-free High-resolution Image Generationcited by 3
  28. 2025SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketchescited by 3
  29. 2026Deep Sketch-Based 3D Modeling: A Surveycited by 3
  30. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  31. 2025Color Bind: Exploring Color Perception in Text-to-Image Modelscited by 2
  32. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  33. 2025Test-Time Preference Optimization for Image Restorationcited by 2
  34. 2026Momentum Guidance: Plug-and-Play Guidance for Flow Modelscited by 2
  35. 2026DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Modelscited by 2
  36. 2024LTOS: Layout-controllable Text-Object Synthesis via Adaptive Cross-attention Fusionscited by 1
  37. 2024Prompt-Consistency Image Generation (PCIG): A Unified Framework Integrating LLMs, Knowledge Graphs, and Controllable Diffusion Modelscited by 1
  38. 2024DiCTI: Diffusion-based Clothing Designer via Text-guided Inputcited by 1

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  2. 2026MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis
  3. 2024Conditional Image Synthesis with Diffusion Models: A Survey
  4. 2024ECNet: Effective Controllable Text-to-Image Diffusion Models
  5. 2026Efficient Text-to-Image Generation: An Adaptive Step Schedule Controller for Diffusion Models
  6. 2024CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
  7. 2023Nested Diffusion Processes for Anytime Image Generation
  8. 2024TCIG: Two-Stage Controlled Image Generation with Quality Enhancement through Diffusion