mlsanityData to 14 Oct 2026

Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion

arXiv 2303.08767 ↗First posted 15 Mar 2023cs.CV cs.AIcited by 39

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have shown superior performance in image generation and manipulation, but the inherent stochasticity presents challenges in preserving and manipulating image content and identity. While previous approaches like DreamBooth and Textual Inversion have proposed model or latent representation personalization to maintain the content, their reliance on multiple reference images and complex training limits their practicality. In this paper, we present a simple yet highly effective approach to personalization using highly personalized (HiPer) text embedding by decomposing the CLIP embedding space for personalization and content manipulation. Our method does not require model fine-tuning or identifiers, yet still enables manipulation of background, texture, and motion with just a single image and target text. Through experiments on diverse target texts, we demonstrate that our approach produces highly personalized and complex semantic image edits across a wide range of tasks. We believe that the novel understanding of the text embedding space presented in this work has the potential to inspire further research across various tasks.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 42 of its 42 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

42 of 42 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  2. 2022Diffusion-based Image Translation using Disentangled Style and Content Representation30 Sep
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  7. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  9. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
21 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  5. 2018Self-Attention Generative Adversarial Networkscited by 809
  6. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  8. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  9. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  10. 2023Zero-shot Image-to-Image Translationcited by 505
  11. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  12. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  13. 2019Controllable Text-to-Image Generationcited by 187
  14. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  15. 2021UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Modelscited by 133
  16. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  17. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  18. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  19. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  20. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  21. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 21; 2025: 8; 2026: 3.

39 papers.

Sort:

By area:

  1. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  2. 2023LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-Oncited by 152
  3. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  4. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editingcited by 68
  5. 2024Visual Style Prompting with Swapping Self-Attentioncited by 63
  6. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 41
  7. 2023Training-free Content Injection using h-space in Diffusion Modelscited by 36
  8. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  9. 2024ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglementcited by 28
  10. 2024Tuning-Free Image Customization with Image and Text Guidancecited by 25
  11. 2024Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editingcited by 21
  12. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
27 more
  1. 2024Minority-Focused Text-to-Image Generation via Prompt Optimizationcited by 18
  2. 2025Personalize Your LLM: Fake it then Align itcited by 15
  3. 2024PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editingcited by 10
  4. 2024Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifiercited by 10
  5. 2024HARIVO: Harnessing Text-to-Image Models for Video Generationcited by 8
  6. 2025Free-Lunch Color-Texture Disentanglement for Stylized Image Generationcited by 8
  7. 2025TaxaDiffusion: Progressively Trained Diffusion Model for Fine-Grained Species Generationcited by 7
  8. 2024Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidancecited by 6
  9. 2025StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidancecited by 6
  10. 2024Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editingcited by 5
  11. 2024Visual Concept-driven Image Generation with Text-to-Image Diffusion Modelcited by 4
  12. 2024Dream-in-Style: Text-to-3D Generation Using Stylized Score Distillationcited by 4
  13. 2024Prompt-Softbox-Prompt: A Free-Text Embedding Control for Image Editingcited by 4
  14. 2024HypDAE: Hyperbolic Diffusion Autoencoders for Hierarchical Few-shot Image Generationcited by 4
  15. 2026DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generationcited by 3
  16. 2026TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Lifecited by 3
  17. 2024AlignIT: Enhancing Prompt Alignment in Customization of Text-to-Image Modelscited by 2
  18. 2024An Improved Method for Personalizing Diffusion Modelscited by 2
  19. 2024Assessing Open-world Forgetting in Generative Image Model Customizationcited by 2
  20. 2025HF-VTON: High-Fidelity Virtual Try-On via Consistent Geometric and Semantic Alignmentcited by 2
  21. 2023Face Recognition Using Synthetic Face Datacited by 1
  22. 2023DiverseDream: Diverse Text-to-3D Synthesis with Augmented Text Embeddingcited by 1
  23. 2024Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editingcited by 1
  24. 2024Layout-and-Retouch: A Dual-stage Framework for Improving Diversity in Personalized Image Generationcited by 1
  25. 2025Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Imagecited by 1
  26. 2025Training Free Stylized Abstractioncited by 0
  27. 2026DenseControl: Instance-Level Controllable Synthesis of Dense Crowd Imagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning to Customize Text-to-Image Diffusion In Diverse Context
  2. 2026Zero-Shot Personalization of Objects via Textual Inversion
  3. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
  4. 2023PRedItOR: Text Guided Image Editing with Diffusion Prior
  5. 2023CatVersion: Concatenating Embeddings for Diffusion-Based Text-to-Image Personalization
  6. 2022LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models
  7. 2024Uncovering the Text Embedding in Text-to-Image Diffusion Models
  8. 2023Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Models