mlsanityData to 7 Oct 2026

DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

arXiv 2110.02711 ↗First posted 6 Oct 2021cs.CV cs.AI cs.LGcited by 640

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zero-shot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs text-driven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zero-shot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git.

Key facts. Cited by 640 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 60 of its 69 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

60 of 69 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2016Wide Residual Networkscited by 3,057
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2015Variational Inference with Normalizing Flowscited by 2,161
  12. 2016Density estimation using Real NVPcited by 2,029
  13. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  14. 2018Group Normalizationcited by 1,514
  15. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  16. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  17. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  18. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  19. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  20. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  21. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  22. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  23. 2016Pixel Recurrent Neural Networkscited by 814
  24. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  25. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  26. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  27. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  28. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  29. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  30. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  31. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  32. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  33. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  34. 2017Compressed Sensing using Generative Modelscited by 337
  35. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 331
  36. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
  37. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  38. 2017Photographic Image Synthesis with Cascaded Refinement Networkscited by 223

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 49; 2023: 210; 2024: 204; 2025: 113; 2026: 64.

640 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  10. 2022Diffusion Models in Vision: A Surveycited by 1,008
  11. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  12. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
38 more
  1. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  2. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  3. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  4. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  5. 2023Zero-shot Image-to-Image Translationcited by 503
  6. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  7. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  8. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  9. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  10. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 243
  11. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  12. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  13. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  14. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  15. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  16. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  17. 2023Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Modelscited by 172
  18. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  19. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  20. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  21. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  22. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priorscited by 156
  23. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  24. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Modelscited by 155
  25. 2023StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Modelscited by 154
  26. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 148
  27. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  28. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  29. 2023Are Diffusion Models Vulnerable to Membership Inference Attacks?cited by 135
  30. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  31. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  32. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  33. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  34. 2023Delta Denoising Scorecited by 120
  35. 2024Training-free Camera Control for Video Generationcited by 119
  36. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Modelscited by 116
  37. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  38. 2024Transparent Image Layer Diffusion using Latent Transparencycited by 111

50 most cited of 640 All 640 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Image Inversion: A Survey from GANs to Diffusion and Beyond
  2. 2022Blended Latent Diffusion
  3. 2025Beyond Sliders: Mastering the Art of Diffusion-based Image Manipulation
  4. 2024The Curious Case of End Token: A Zero-Shot Disentangled Image Editing using CLIP
  5. 2022Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
  7. 2024Distilling Diffusion Models into Conditional GANs
  8. 2023NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models