# StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing

arXiv: https://arxiv.org/abs/2303.15649 · HTML: https://mlsanity.com/paper/2303.15649/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 61 of its 85 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)

## Built on by (most cited first, 30 of 68)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [Improving Tuning-Free Real Image Editing with Proximal Guidance](https://mlsanity.com/paper/2306.05414/) (2023)
- [Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing](https://mlsanity.com/paper/2309.15664/) (2023)
- [Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2402.05375/) (2024)
- [KV-Edit: Training-Free Image Editing for Precise Background Preservation](https://mlsanity.com/paper/2502.17363/) (2025)
- [CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation](https://mlsanity.com/paper/2310.13165/) (2023)
- [Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.07132/) (2024)
- [Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference](https://mlsanity.com/paper/2312.09608/) (2023)
- [FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing](https://mlsanity.com/paper/2407.17850/) (2024)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion](https://mlsanity.com/paper/2412.04301/) (2024)
- [StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer](https://mlsanity.com/paper/2501.11319/) (2025)
- [Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models](https://mlsanity.com/paper/2403.11105/) (2024)
- [IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models](https://mlsanity.com/paper/2311.06792/) (2023)
- [MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance](https://mlsanity.com/paper/2312.11396/) (2023)
- [h-Edit: Effective and Flexible Diffusion-Based Editing via Doob's h-Transform](https://mlsanity.com/paper/2503.02187/) (2025)
- [Z^*: Zero-shot Style Transfer via Attention Rearrangement](https://mlsanity.com/paper/2311.16491/) (2023)
- [Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps](https://mlsanity.com/paper/2406.14539/) (2024)
- [CLIPDrag: Combining Text-based and Drag-based Instructions for Image Editing](https://mlsanity.com/paper/2410.03097/) (2024)
- [Wavelet-Guided Acceleration of Text Inversion in Diffusion-Based Image Editing](https://mlsanity.com/paper/2401.09794/) (2024)
- [LocInv: Localization-aware Inversion for Text-Guided Image Editing](https://mlsanity.com/paper/2405.01496/) (2024)
- [Music Style Transfer with Time-Varying Inversion of Diffusion Models](https://mlsanity.com/paper/2402.13763/) (2024)
- [Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifier](https://mlsanity.com/paper/2410.22317/) (2024)
- [Anchor Token Matching: Implicit Structure Locking for Training-free AR Image Editing](https://mlsanity.com/paper/2504.10434/) (2025)
- [Manipulating Embeddings of Stable Diffusion Prompts](https://mlsanity.com/paper/2308.12059/) (2023)
- [The Art of Deception: Color Visual Illusions and Diffusion Models](https://mlsanity.com/paper/2412.10122/) (2024)
- [AttenST: A Training-Free Attention-Driven Style Transfer Framework with Pre-Trained Diffusion Models](https://mlsanity.com/paper/2503.07307/) (2025)
- [IterInv: Iterative Inversion for Pixel-Level T2I Models](https://mlsanity.com/paper/2310.19540/) (2023)
- [FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model](https://mlsanity.com/paper/2311.14926/) (2023)

## Abstract (arXiv)

A significant research effort is focused on exploiting the amazing capacities of pretrained diffusion models for the editing of images.They either finetune the model, or invert the image in the latent space of the pretrained model. However, they suffer from two problems: (1) Unsatisfying results for selected regions and unexpected changes in non-selected regions.(2) They require careful text prompt editing where the prompt should include all visual objects in the input image.To address this, we propose two improvements: (1) Only optimizing the input of the value linear network in the cross-attention layers is sufficiently powerful to reconstruct a real image. (2) We propose attention regularization to preserve the object-like attention maps after reconstruction and editing, enabling us to obtain accurate style editing without invoking significant structural changes. We further improve the editing technique that is used for the unconditional branch of classifier-free guidance as used by P2P. Extensive experimental prompt-editing results on a variety of images demonstrate qualitatively and quantitatively that our method has superior editing capabilities compared to existing and concurrent works. See our accompanying code in Stylediffusion: urlhttps://github.com/sen-mao/StyleDiffusion.