# Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer

arXiv: https://arxiv.org/abs/2303.08622 · HTML: https://mlsanity.com/paper/2303.08622/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 36 of its 44 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)

## Built on by (most cited first, 30 of 59)

- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models](https://mlsanity.com/paper/2305.16225/) (2023)
- [StyleShot: A Snapshot on Any Style](https://mlsanity.com/paper/2407.01414/) (2024)
- [Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing](https://mlsanity.com/paper/2311.18608/) (2023)
- [Towards Accurate Post-training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.18723/) (2023)
- [NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models](https://mlsanity.com/paper/2312.05390/) (2023)
- [Language-driven Grasp Detection](https://mlsanity.com/paper/2406.09489/) (2024)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [Any-to-Any Style Transfer: Making Picasso and Da Vinci Collaborate](https://mlsanity.com/paper/2304.09728/) (2023)
- [InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser](https://mlsanity.com/paper/2311.15040/) (2023)
- [Z^*: Zero-shot Style Transfer via Attention Rearrangement](https://mlsanity.com/paper/2311.16491/) (2023)
- [Free-Editor: Zero-shot Text-driven 3D Scene Editing](https://mlsanity.com/paper/2312.13663/) (2023)
- [GDA: Generalized Diffusion for Robust Test-time Adaptation](https://mlsanity.com/paper/2404.00095/) (2024)
- [Using Diffusion Models to Generate Synthetic Labelled Data for Medical Image Segmentation](https://mlsanity.com/paper/2310.16794/) (2023)
- [Portrait Diffusion: Training-free Face Stylization with Chain-of-Painting](https://mlsanity.com/paper/2312.02212/) (2023)
- [Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization](https://mlsanity.com/paper/2401.16762/) (2024)
- [AttenST: A Training-Free Attention-Driven Style Transfer Framework with Pre-Trained Diffusion Models](https://mlsanity.com/paper/2503.07307/) (2025)
- [QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation](https://mlsanity.com/paper/2507.04599/) (2025)
- [Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation](https://mlsanity.com/paper/2312.08195/) (2023)
- [High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion](https://mlsanity.com/paper/2410.11838/) (2024)
- [Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors](https://mlsanity.com/paper/2512.17376/) (2025)
- [StyleMamba : State Space Model for Efficient Text-driven Image Style Transfer](https://mlsanity.com/paper/2405.05027/) (2024)
- [Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training](https://mlsanity.com/paper/2407.08946/) (2024)
- [DiffuseST: Unleashing the Capability of the Diffusion Model for Style Transfer](https://mlsanity.com/paper/2410.15007/) (2024)
- [Balanced Image Stylization with Style Matching Score](https://mlsanity.com/paper/2503.07601/) (2025)
- [MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping](https://mlsanity.com/paper/2604.08364/) (2026)
- [Dequantization and Color Transfer with Diffusion Models](https://mlsanity.com/paper/2307.02698/) (2023)
- [SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer](https://mlsanity.com/paper/2503.04119/) (2025)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)

## Abstract (arXiv)

Diffusion models have shown great promise in text-guided image style transfer, but there is a trade-off between style transformation and content preservation due to their stochastic nature. Existing methods require computationally expensive fine-tuning of diffusion models or additional neural network. To address this, here we propose a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks. By leveraging patch-wise contrastive loss between generated samples and original image embeddings in the pre-trained diffusion model, our method can generate images with the same semantic content as the source image in a zero-shot manner. Our approach outperforms existing methods while preserving content and requiring no additional training, not only for image style transfer but also for image-to-image translation and manipulation. Our experimental results validate the effectiveness of our proposed method.