mlsanityData to 14 Oct 2026

Diffusion-based Image Translation using Disentangled Style and Content Representation

arXiv 2209.15264 ↗First posted 30 Sep 2022cs.CV cs.AI cs.LGcited by 149

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion-based image translation guided by semantic texts or a single target image has enabled flexible style transfer which is not limited to the specific domains. Unfortunately, due to the stochastic nature of diffusion models, it is often difficult to maintain the original content of the image during the reverse diffusion. To address this, here we present a novel diffusion-based unsupervised image translation method using disentangled style and content representation. Specifically, inspired by the splicing Vision Transformer, we extract intermediate keys of multihead self attention layer from ViT model and used them as the content preservation loss. Then, an image guided style transfer is performed by matching the [CLS] classification token from the denoised samples and target image, whereas additional CLIP loss is used for the text-driven style transfer. To further accelerate the semantic change during the reverse diffusion, we also propose a novel semantic divergence loss and resampling strategy. Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.

Key facts. Cited by 149 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 56 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

46 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  4. 2021Palette: Image-to-Image Diffusion Models10 Nov
  5. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
29 more
  1. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  3. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 639
  4. 2021Image Segmentation Using Text and Image Promptscited by 512
  5. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 508
  6. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  7. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  8. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 307
  9. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  10. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  11. 2021Few-shot Image Generation via Cross-domain Correspondencecited by 207
  12. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  13. 2018Arbitrary Style Transfer with Style-Attentional Networkscited by 198
  14. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 168
  15. 2019Photorealistic Style Transfer via Wavelet Transformscited by 166
  16. 2021UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Modelscited by 133
  17. 2021Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Imagescited by 113
  18. 2021HairCLIP: Design Your Hair by Text and Reference Imagecited by 80
  19. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  20. 2021Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networkscited by 68
  21. 2021Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Modelscited by 51
  22. 2021JoJoGAN: One Shot Face Stylizationcited by 49
  23. 2022Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Taskscited by 42
  24. 2017ComboGAN: Unrestrained Scalability for Image Domain Translationcited by 37
  25. 2019CollaGAN : Collaborative GAN for Missing Image Data Imputationcited by 33
  26. 2022One-Shot Adaptation of GAN in Just One CLIPcited by 31
  27. 2022FlexIT: Towards Flexible Semantic Image Translationcited by 28
  28. 2020TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Imagescited by 23
  29. 2020PlenoptiCam v1.0: A light-field imaging frameworkcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 48; 2024: 51; 2025: 33; 2026: 12.

149 papers.

Sort:

By area:

  1. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  2. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  3. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  4. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  5. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  6. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 196
  7. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 165
  8. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  9. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  10. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 110
  11. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  12. 2023Diffusion Models for Imperceptible and Transferable Adversarial Attackcited by 94
38 more
  1. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  2. 2023Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusioncited by 78
  3. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editingcited by 76
  4. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
  5. 2022MagicMix: Semantic Mixing with Diffusion Modelscited by 63
  6. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  7. 2023Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfercited by 59
  8. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  9. 2023Relightful Harmonization: Lighting-aware Portrait Background Replacementcited by 57
  10. 2023Contrastive Denoising Score for Text-guided Latent Diffusion Image Editingcited by 49
  11. 2023WaveDM: Wavelet-Based Diffusion Models for Image Restorationcited by 47
  12. 2024Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignmentcited by 43
  13. 2023ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptationcited by 40
  14. 2023Highly Personalized Text Embedding for Image Manipulation by Stable Diffusioncited by 39
  15. 2022High-Fidelity Guided Image Synthesis with Latent Diffusion Modelscited by 36
  16. 2023Training-free Content Injection using h-space in Diffusion Modelscited by 36
  17. 2023ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bankcited by 36
  18. 2023DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Modelscited by 33
  19. 2024DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protectioncited by 32
  20. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 29
  21. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  22. 2024DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editingcited by 28
  23. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 28
  24. 2023Diff-Privacy: Diffusion-based Face Privacy Protectioncited by 26
  25. 2024Noise Map Guidance: Inversion with Spatial Context for Real Image Editingcited by 25
  26. 2025StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfercited by 23
  27. 2023Exploring the Naturalness of AI-Generated Imagescited by 22
  28. 2023MagiCapture: High-Resolution Multi-Concept Portrait Customizationcited by 20
  29. 2023MDP: A Generalized Framework for Text-Guided Image Editing by Manipulating the Diffusion Pathcited by 18
  30. 2023Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysiscited by 18
  31. 2023ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusioncited by 16
  32. 2024One-Shot Structure-Aware Stylized Image Synthesiscited by 14
  33. 2024TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generationcited by 14
  34. 2025ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translationcited by 14
  35. 2024UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Texturescited by 13
  36. 2023Painterly Image Harmonization using Diffusion Modelcited by 12
  37. 2024InstructGIE: Towards Generalizable Image Editingcited by 12
  38. 2024Diffusion-driven GAN Inversion for Multi-Modal Face Image Generationcited by 12

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models
  2. 2024FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models
  3. 2025Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
  4. 2024DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations
  5. 2024ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model
  6. 2024WikiStyle+: A Multimodal Approach to Content-Style Representation Disentanglement for Artistic Image Stylization
  7. 2025Leveraging Diffusion Models for Stylization using Multiple Style Images
  8. 2023Z^*: Zero-shot Style Transfer via Attention Rearrangement