mlsanityData to 14 Oct 2026

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

arXiv 2302.12228 ↗First posted 23 Feb 2023cs.CV cs.GR cs.LGcited by 203

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts. However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity. To overcome these limitations, we propose an encoder-based domain-tuning approach. Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain. Specifically, we employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain, e.g. a specific face, and learns to map it into a word-embedding representing the concept. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts. Together, these components are used to guide the learning of unseen concepts, allowing us to personalize a model using only a single image and as few as 5 training steps - accelerating personalization from dozens of minutes to seconds, while preserving quality.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 72 of its 93 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

72 of 93 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models8 Feb
  2. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
38 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  6. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  7. 2018On the Spectral Bias of Neural Networkscited by 1,341
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2017Deep Image Priorcited by 1,098
  10. 2016HyperNetworkscited by 1,027
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  12. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  13. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  14. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  15. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  16. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  17. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  18. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  19. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  20. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  21. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  22. 2020Closed-Form Factorization of Latent Semantics in GANscited by 408
  23. 2021GAN Inversion: A Surveycited by 367
  24. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  25. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  26. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  27. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  28. 2020CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognitioncited by 321
  29. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  30. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  31. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  32. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  33. 2021ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinementcited by 248
  34. 2021Personalized Federated Learning using Hypernetworkscited by 212
  35. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  36. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  37. 2019Image Processing Using Multi-Code GAN Priorcited by 185
  38. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 50; 2024: 95; 2025: 48; 2026: 10.

203 papers.

Sort:

By area:

  1. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  2. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  3. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  4. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  5. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  6. 2023Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Modelscited by 174
  7. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 143
  8. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  9. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  10. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 126
  11. 2024Training-Free Consistent Text-to-Image Generationcited by 125
  12. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
38 more
  1. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
  2. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  3. 2024Arc2Face: A Foundation Model for ID-Consistent Human Facescited by 101
  4. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  5. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  6. 2023Face0: Instantaneously Conditioning a Text-to-Image Model on a Facecited by 78
  7. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
  8. 2023DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generationcited by 75
  9. 2023ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Modelscited by 75
  10. 2023PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Modelscited by 70
  11. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editingcited by 68
  12. 2025ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuningcited by 68
  13. 2025Multi-subject Open-set Personalization in Video Generationcited by 66
  14. 2024OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Modelscited by 64
  15. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 63
  16. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  17. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  18. 2023FaceStudio: Put Your Face Everywhere in Secondscited by 53
  19. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  20. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
  21. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 50
  22. 2024FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translationcited by 49
  23. 2024FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Compositioncited by 49
  24. 2024LCM-Lookahead for Encoder-based Text-to-Image Personalizationcited by 48
  25. 2023Photoswap: Personalized Subject Swapping in Imagescited by 47
  26. 2023Customizing 360-Degree Panoramas through Text-to-Image Diffusion Modelscited by 47
  27. 2024UniPortrait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalizationcited by 47
  28. 2024Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Modelcited by 45
  29. 2024MultiBooth: Towards Generating All Your Concepts in an Image from Textcited by 45
  30. 2023ControlCom: Controllable Image Composition using Diffusion Modelcited by 43
  31. 2023Single-Image 3D Human Digitization with Shape-Guided Diffusioncited by 42
  32. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  33. 2024Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigmcited by 39
  34. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  35. 2024Customizing Text-to-Image Models with a Single Image Paircited by 36
  36. 2023MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentationcited by 35
  37. 2024PALP: Prompt Aligned Personalization of Text-to-Image Modelscited by 35
  38. 2024DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Modelscited by 34

50 most cited of 203 All 203 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Steering Guidance for Personalized Text-to-Image Diffusion Models
  2. 2025Finetuning-Free Personalization of Text to Image Generation via Hypernetworks
  3. 2026Zero-Shot Personalization of Objects via Textual Inversion
  4. 2025Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models
  5. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
  6. 2025Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter
  7. 2024Learning to Customize Text-to-Image Diffusion In Diverse Context
  8. 2024TextBoost: Boosting Text Encoder for Personalized Text-to-Image Generation