mlsanityData to 14 Oct 2026

More Control for Free! Image Synthesis with Semantic Diffusion Guidance

arXiv 2112.05744 ↗First posted 10 Dec 2021cs.CV cs.GRcited by 212

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from a reference image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffusion model using the gradient of image-text or image matching scores, without re-training the diffusion model. We explore CLIP-based language guidance as well as both content and style-based image guidance in a unified framework. Our text-guided synthesis approach can be applied to datasets without associated text annotations. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained text-guided image synthesis, synthesis of images related to a style or content reference image, and examples with both textual and image guidance.

Key facts. Cited by 212 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

51 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
38 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  2. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  3. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  4. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  5. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  6. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  8. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  13. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  14. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  15. 2017Universal Style Transfer via Feature Transformscited by 354
  16. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  17. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  18. 2019Controllable Text-to-Image Generationcited by 187
  19. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178
  20. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  21. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 168
  22. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  23. 2021Instance-Conditioned GANcited by 121
  24. 2016Learning What and Where to Drawcited by 105
  25. 2016Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesiscited by 102
  26. 2016Controlling Perceptual Factors in Neural Style Transfercited by 100
  27. 2017Multi-Content GAN for Few-Shot Font Style Transfercited by 100
  28. 2021GAN-Control: Explicitly Controllable GANscited by 91
  29. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  30. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  31. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  32. 2018Arbitrary Style Transfer with Deep Feature Reshufflecited by 75
  33. 2017Language-Based Image Editing with Recurrent Attentive Modelscited by 53
  34. 2018ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributescited by 50
  35. 2017Be Your Own Prada: Fashion Synthesis with Structural Coherencecited by 49
  36. 2019Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networkscited by 8
  37. 2018C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesiscited by 6
  38. 2018Language Guided Fashion Image Manipulation with Feature-wise Transformationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 44; 2023: 62; 2024: 64; 2025: 28; 2026: 13.

212 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  3. 2022Diffusion Models in Vision: A Surveycited by 1,015
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  5. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  6. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  7. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  8. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  9. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 301
  10. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 294
  11. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  12. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
38 more
  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  3. 2023Generative Diffusion Prior for Unified Image Restoration and Enhancementcited by 199
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  5. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  6. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  7. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  8. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  9. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Viewscited by 148
  10. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
  11. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  12. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 125
  13. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Modelscited by 116
  14. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  15. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 109
  16. 2022Diffusion Visual Counterfactual Explanationscited by 96
  17. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoningcited by 92
  18. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 82
  19. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidancecited by 77
  20. 2022Back to the Source: Diffusion-Driven Test-Time Adaptationcited by 71
  21. 2022Conversion Between CT and MRI Images Using Diffusion and Score-Matching Modelscited by 69
  22. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  23. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
  24. 2023Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Modelcited by 60
  25. 2023Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfercited by 59
  26. 2024Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidancecited by 59
  27. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Modelscited by 54
  28. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 54
  29. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  30. 2022Efficient Diffusion Models for Vision: A Surveycited by 51
  31. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generationcited by 50
  32. 2023WaveDM: Wavelet-Based Diffusion Models for Image Restorationcited by 47
  33. 2023DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly Detectioncited by 46
  34. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  35. 2022UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Imagecited by 42
  36. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Modelcited by 41
  37. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  38. 2022Diff-Font: Diffusion Model for Robust One-Shot Font Generationcited by 40

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance
  2. 2023Filter-Guided Diffusion for Controllable Image Generation
  3. 2022Semantic Image Synthesis via Diffusion Models
  4. 2024ECNet: Effective Controllable Text-to-Image Diffusion Models
  5. 2024Diffusion-Based Conditional Image Editing through Optimized Inference with Guidance
  6. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
  8. 2023Semantic Guidance Tuning for Text-To-Image Diffusion Models