mlsanityData to 14 Oct 2026

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

arXiv 2211.12572 ↗First posted 22 Nov 2022cs.CV cs.AIcited by 906

Not on the map. Why?

Cited by 542 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale text-to-image generative models have been a revolutionary breakthrough in the evolution of generative AI, allowing us to synthesize diverse images that convey highly complex visual concepts. However, a pivotal challenge in leveraging such models for real-world content creation tasks is providing users with control over the generated content. In this paper, we present a new framework that takes text-to-image synthesis to the realm of image-to-image translation -- given a guidance image and a target text prompt, our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text, while preserving the semantic layout of the source image. Specifically, we observe and empirically demonstrate that fine-grained control over the generated structure can be achieved by manipulating spatial features and their self-attention inside the model. This results in a simple and effective approach, where features extracted from the guidance image are directly injected into the generation process of the target image, requiring no training or fine-tuning and applicable for both real or generated guidance images. We demonstrate high-quality results on versatile text-guided image translation tasks, including translating sketches, rough drawings and animations into realistic images, changing of the class and appearance of objects in a given image, and modifications of global qualities such as lighting and color.

Key facts. Cited by 906 papers in the map (data through 14 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

44 of 54 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
32 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  8. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  9. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  10. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  11. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  12. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 639
  13. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  14. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  15. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  16. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  17. 2022Blended Latent Diffusioncited by 432
  18. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  19. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  20. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  21. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  22. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  23. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 168
  24. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  25. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  26. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 149
  27. 2021Paint by Wordcited by 107
  28. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  29. 2022FlexIT: Towards Flexible Semantic Image Translationcited by 28
  30. 2017Smart, Sparse Contours to Represent and Edit Imagescited by 19
  31. 2022A Style-aware Discriminator for Controllable Image Translationcited by 16
  32. 2020Image Shape Manipulation from a Single Augmented Training Samplecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 180; 2024: 274; 2025: 274; 2026: 176.

906 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  3. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  4. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  5. 2023Zero-shot Image-to-Image Translationcited by 505
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  7. 2023Emergent Correspondence from Image Diffusioncited by 446
  8. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  9. 2024OmniGen: Unified Image Generationcited by 394
  10. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  11. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 324
  12. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
38 more
  1. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  2. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  3. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  4. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 304
  5. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  6. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  7. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  8. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  9. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  10. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  11. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  12. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 233
  13. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  14. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 217
  15. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  16. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 214
  17. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  18. 2023A Survey on Video Diffusion Modelscited by 206
  19. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  20. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
  21. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  22. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 196
  23. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
  24. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 183
  25. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  26. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  27. 2023Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Modelscited by 174
  28. 2024InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generationcited by 170
  29. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  30. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 165
  31. 2023Dense Text-to-Image Generation with Attention Modulationcited by 163
  32. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  33. 2024OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-oncited by 151
  34. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  35. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  36. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  37. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 142
  38. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139

50 most cited of 906 All 906 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Implementing and Experimenting with Diffusion Models for Text-to-Image Generation
  2. 2024Conditional Text-to-Image Generation with Reference Guidance
  3. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
  4. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models
  5. 2023SEGA: Instructing Text-to-Image Models using Semantic Guidance
  6. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation
  7. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering
  8. 2023Design Booster: A Text-Guided Diffusion Model for Image Translation with Spatial Layout Preservation