# Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach

arXiv: https://arxiv.org/abs/2305.13579 · HTML: https://mlsanity.com/paper/2305.13579/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Shifted Diffusion for Text-to-image Generation](https://mlsanity.com/paper/2211.15388/) (2022)

## Built on by (most cited first, 30 of 43)

- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning](https://mlsanity.com/paper/2307.11410/) (2023)
- [VideoBooth: Diffusion-based Video Generation with Image Prompts](https://mlsanity.com/paper/2312.00777/) (2023)
- [Arc2Face: A Foundation Model for ID-Consistent Human Faces](https://mlsanity.com/paper/2403.11641/) (2024)
- [Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models](https://mlsanity.com/paper/2307.06925/) (2023)
- [PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2309.05793/) (2023)
- [OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models](https://mlsanity.com/paper/2403.10983/) (2024)
- [FaceStudio: Put Your Face Everywhere in Seconds](https://mlsanity.com/paper/2312.02663/) (2023)
- [PALP: Prompt Aligned Personalization of Text-to-Image Models](https://mlsanity.com/paper/2401.06105/) (2024)
- [IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models](https://mlsanity.com/paper/2403.13535/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Control](https://mlsanity.com/paper/2408.05083/) (2024)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [NewMove: Customizing text-to-video models with novel motions](https://mlsanity.com/paper/2312.04966/) (2023)
- [InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser](https://mlsanity.com/paper/2311.15040/) (2023)
- [Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance](https://mlsanity.com/paper/2501.05379/) (2025)
- [T-LoRA: Single Image Diffusion Model Customization Without Overfitting](https://mlsanity.com/paper/2507.05964/) (2025)
- [CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization](https://mlsanity.com/paper/2408.15914/) (2024)
- [Data Synthesis with Diverse Styles for Face Recognition via 3DMM-Guided Diffusion](https://mlsanity.com/paper/2504.00430/) (2025)
- [Customization Assistant for Text-to-image Generation](https://mlsanity.com/paper/2312.03045/) (2023)
- [Group and Shuffle: Efficient Structured Orthogonal Parametrization](https://mlsanity.com/paper/2406.10019/) (2024)
- [Text2Place: Affordance-aware Text Guided Human Placement](https://mlsanity.com/paper/2407.15446/) (2024)
- [ID-Patch: Robust ID Association for Group Photo Personalization](https://mlsanity.com/paper/2411.13632/) (2024)
- [Cross Initialization for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2312.15905/) (2023)
- [AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation](https://mlsanity.com/paper/2406.05000/) (2024)
- [Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting](https://mlsanity.com/paper/2404.14007/) (2024)
- [SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation](https://mlsanity.com/paper/2506.23690/) (2025)
- [HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generation](https://mlsanity.com/paper/2410.08192/) (2024)
- [Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generation](https://mlsanity.com/paper/2508.08949/) (2025)
- [Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation](https://mlsanity.com/paper/2406.09305/) (2024)

## Abstract (arXiv)

Recent text-to-image generation models have demonstrated impressive capability of generating text-aligned images with high fidelity. However, generating images of novel concept provided by the user input image is still a challenging task. To address this problem, researchers have been exploring various methods for customizing pre-trained text-to-image generation models. Currently, most existing methods for customizing pre-trained text-to-image generation models involve the use of regularization techniques to prevent over-fitting. While regularization will ease the challenge of customization and leads to successful content creation with respect to text guidance, it may restrict the model capability, resulting in the loss of detailed information and inferior performance. In this work, we propose a novel framework for customized text-to-image generation without the use of regularization. Specifically, our proposed framework consists of an encoder network and a novel sampling method which can tackle the over-fitting problem without the use of regularization. With the proposed framework, we are able to customize a large-scale text-to-image generation model within half a minute on single GPU, with only one image provided by the user. We demonstrate in experiments that our proposed framework outperforms existing methods, and preserves more fine-grained details.