# Inversion-Based Style Transfer with Diffusion Models

arXiv: https://arxiv.org/abs/2211.13203 · HTML: https://mlsanity.com/paper/2211.13203/

## Key facts

- Cited by 291 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 47 of its 60 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [Arbitrary Style Transfer with Style-Attentional Networks](https://mlsanity.com/paper/1812.02342/) (2018)
- [StyTr2: Image Style Transfer with Transformers](https://mlsanity.com/paper/2105.14576/) (2021)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer](https://mlsanity.com/paper/2108.03647/) (2021)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)

## Built on by (most cited first, 30 of 291)

- [Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization](https://mlsanity.com/paper/2308.14469/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [StyleDrop: Text-to-Image Generation in Any Style](https://mlsanity.com/paper/2306.00983/) (2023)
- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On](https://mlsanity.com/paper/2312.01725/) (2023)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing](https://mlsanity.com/paper/2407.05600/) (2024)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [Consistent Video-to-Video Transfer Using Synthetic Dataset](https://mlsanity.com/paper/2311.00213/) (2023)
- [An Undetectable Watermark for Generative Image Models](https://mlsanity.com/paper/2410.07369/) (2024)
- [CSGO: Content-Style Composition in Text-to-Image Generation](https://mlsanity.com/paper/2408.16766/) (2024)
- [FocalDreamer: Text-driven 3D Editing via Focal-fusion Assembly](https://mlsanity.com/paper/2308.10608/) (2023)
- [ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models](https://mlsanity.com/paper/2305.16225/) (2023)
- [DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations](https://mlsanity.com/paper/2403.06951/) (2024)
- [ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion](https://mlsanity.com/paper/2310.10343/) (2023)
- [StyleShot: A Snapshot on Any Style](https://mlsanity.com/paper/2407.01414/) (2024)
- [Insert Anything: Image Insertion via In-Context Editing in DiT](https://mlsanity.com/paper/2504.15009/) (2025)
- [CRS-Diff: Controllable Remote Sensing Image Generation with Diffusion Model](https://mlsanity.com/paper/2403.11614/) (2024)
- [Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks](https://mlsanity.com/paper/2403.00644/) (2024)
- [K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs](https://mlsanity.com/paper/2502.18461/) (2025)
- [WaveDM: Wavelet-Based Diffusion Models for Image Restoration](https://mlsanity.com/paper/2305.13819/) (2023)
- [RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model](https://mlsanity.com/paper/2309.00810/) (2023)
- [On Exact Inversion of DPM-Solvers](https://mlsanity.com/paper/2311.18387/) (2023)
- [StyleMaster: Stylize Your Video with Artistic Generation and Translation](https://mlsanity.com/paper/2412.07744/) (2024)
- [SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation](https://mlsanity.com/paper/2403.16605/) (2024)
- [ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors](https://mlsanity.com/paper/2311.05463/) (2023)
- [ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation](https://mlsanity.com/paper/2312.02109/) (2023)
- [StyleAdapter: A Unified Stylized Image Generation Model](https://mlsanity.com/paper/2309.01770/) (2023)
- [VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning](https://mlsanity.com/paper/2504.07960/) (2025)
- [Exploiting Diffusion Prior for Generalizable Dense Prediction](https://mlsanity.com/paper/2311.18832/) (2023)

## Abstract (arXiv)

The artistic style within a painting is the means of expression, which includes not only the painting material, colors, and brushstrokes, but also the high-level attributes including semantic elements, object shapes, etc. Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. The pre-trained text-to-image synthesis diffusion probabilistic models have achieved remarkable quality, but it often requires extensive textual descriptions to accurately portray attributes of a particular painting. We believe that the uniqueness of an artwork lies precisely in the fact that it cannot be adequately explained with normal language. Our key idea is to learn artistic style directly from a single painting and then guide the synthesis without providing complex textual descriptions. Specifically, we assume style as a learnable textual description of a painting. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image, thus capturing and transferring the artistic style of a painting. We demonstrate the quality and efficiency of our method on numerous paintings of various artists and styles. Code and models are available at https://github.com/zyxElsa/InST.