mlsanityData to 7 Oct 2026

Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models

arXiv 2307.06925 ↗First posted 13 Jul 2023cs.CV cs.GR cs.LGcited by 85

Not on the map. Why?

Cited by 45 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-to-image (T2I) personalization allows users to guide the creative image generation process by combining their own visual concepts in natural language prompts. Recently, encoder-based techniques have emerged as a new effective approach for T2I personalization, reducing the need for multiple images and long training times. However, most existing encoders are limited to a single-class domain, which hinders their ability to handle diverse concepts. In this work, we propose a domain-agnostic method that does not require any specialized dataset or prior information about the personalized concepts. We introduce a novel contrastive-based regularization technique to maintain high fidelity to the target concept characteristics while keeping the predicted embeddings close to editable regions of the latent space, by pushing the predicted tokens toward their nearest existing CLIP tokens. Our experimental results demonstrate the effectiveness of our approach and show how the learned tokens are more semantic than tokens predicted by unregularized models. This leads to a better representation that achieves state-of-the-art performance while being more flexible than previous methods.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 62 of its 77 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

62 of 77 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  8. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  9. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  10. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  12. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  13. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  14. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  15. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  16. 2023Zero-shot Image-to-Image Translationcited by 503
  17. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  18. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  19. 2022Blended Latent Diffusioncited by 430
  20. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  21. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  22. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  23. 2021GAN Inversion: A Surveycited by 366
  24. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  25. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  26. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  27. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  28. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  29. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  30. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  31. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  32. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  33. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  34. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  35. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  36. 2021Personalized Federated Learning using Hypernetworkscited by 210
  37. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  38. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 48; 2025: 21; 2026: 6.

85 papers.

Sort:

By area:

  1. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  2. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  3. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  4. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  5. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  6. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  7. 2025Multi-subject Open-set Personalization in Video Generationcited by 65
  8. 2024OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Modelscited by 64
  9. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  10. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  11. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  12. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
38 more
  1. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 49
  2. 2024LCM-Lookahead for Encoder-based Text-to-Image Personalizationcited by 48
  3. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  4. 2024Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Modelcited by 44
  5. 2024Customizing Text-to-Image Models with a Single Image Paircited by 36
  6. 2024PALP: Prompt Aligned Personalization of Text-to-Image Modelscited by 35
  7. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  8. 2024BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Modelscited by 27
  9. 2024Personalized Visual Instruction Tuningcited by 24
  10. 2024DiffUHaul: A Training-Free Method for Object Dragging in Imagescited by 22
  11. 2024Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Modelscited by 22
  12. 2025Nested Attention: Semantic-aware Attention Values for Concept Personalizationcited by 21
  13. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  14. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  15. 2023A Data Perspective on Enhanced Identity Preservation for Diffusion Personalizationcited by 18
  16. 2024Magic Insert: Style-Aware Drag-and-Dropcited by 17
  17. 2024ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generationcited by 16
  18. 2024OneActor: Consistent Character Generation via Cluster-Conditioned Guidancecited by 15
  19. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
  20. 2024Fast Personalized Text-to-Image Syntheses With Attention Injectioncited by 14
  21. 2025Object-level Visual Prompts for Compositional Image Generationcited by 14
  22. 2025IP-Composer: Semantic Composition of Visual Conceptscited by 14
  23. 2024Customizing Text-to-Image Diffusion with Object Viewpoint Controlcited by 13
  24. 2024GroundingBooth: Grounding Text-to-Image Customizationcited by 13
  25. 2023Language-Informed Visual Concept Learningcited by 12
  26. 2024An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representationcited by 11
  27. 2024Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learningcited by 11
  28. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  29. 2023SingleInsert: Inserting New Concepts from a Single Image into Text-to-Image Models for Flexible Editingcited by 10
  30. 2023Cross Initialization for Personalized Text-to-Image Generationcited by 10
  31. 2024MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integrationcited by 10
  32. 2024InstructBrush: Learning Attention-based Instruction Optimization for Image Editingcited by 10
  33. 2024AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generationcited by 10
  34. 2024PostEdit: Posterior Sampling for Efficient Zero-Shot Image Editingcited by 9
  35. 2024Personalized Representation from Personalized Generationcited by 9
  36. 2025OmniVTON: Training-Free Universal Virtual Try-Oncited by 9
  37. 2023Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generationcited by 7
  38. 2024MagicFace: Training-free Universal-Style Human Image Customized Synthesiscited by 7

50 most cited of 85 All 85 (JSON)