# T-LoRA: Single Image Diffusion Model Customization Without Overfitting

arXiv: https://arxiv.org/abs/2507.05964 · HTML: https://mlsanity.com/paper/2507.05964/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding](https://mlsanity.com/paper/2312.04461/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [StyleDrop: Text-to-Image Generation in Any Style](https://mlsanity.com/paper/2306.00983/) (2023)
- [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023)
- [Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA](https://mlsanity.com/paper/2304.06027/) (2023)
- [Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models](https://mlsanity.com/paper/2307.06925/) (2023)
- [DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation](https://mlsanity.com/paper/2305.03374/) (2023)
- [AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration](https://mlsanity.com/paper/2309.10438/) (2023)
- [DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization](https://mlsanity.com/paper/2402.09812/) (2024)
- [TokenVerse: Versatile Multi-concept Personalization in Token Modulation Space](https://mlsanity.com/paper/2501.12224/) (2025)

## Built on by (most cited first, 15 of 15)

- [UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios](https://mlsanity.com/paper/2511.18050/) (2025)
- [TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control](https://mlsanity.com/paper/2510.09561/) (2025)
- [Class-specific diffusion models improve military object detection in a low-data domain](https://mlsanity.com/paper/2604.18076/) (2026)
- [Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation](https://mlsanity.com/paper/2603.21884/) (2026)
- [OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models](https://mlsanity.com/paper/2604.05183/) (2026)
- [AnyMS: Bottom-up Attention Decoupling for Layout-guided and Training-free Multi-subject Customization](https://mlsanity.com/paper/2512.23537/) (2025)
- [FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models](https://mlsanity.com/paper/2607.05711/) (2026)
- [Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration](https://mlsanity.com/paper/2601.06605/) (2026)
- [Let Triggers Control: Frequency-Aware Dropout for Effective Token Control](https://mlsanity.com/paper/2603.27199/) (2026)
- [LoRA-Diffusion: Parameter-Efficient Fine-Tuning via Low-Rank Trajectory Decomposition](https://mlsanity.com/paper/2608.12328/) (2026)
- [Understanding LoRA Rank Trade-offs in Diffusion Model Fine-Tuning](https://mlsanity.com/paper/2609.10656/) (2026)
- [Principal-timestep Restricted Init via Sparse Matrix-decomposition in Flow-matching](https://mlsanity.com/paper/2609.15643/) (2026)
- [LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models](https://mlsanity.com/paper/2609.25884/) (2026)
- [Loop Dropout: Regularizing Shared Updates in Looped Language Models](https://mlsanity.com/paper/2609.34218/) (2026)
- [Persistent Watermarking of Text-to-Image Models](https://mlsanity.com/paper/2609.39024/) (2026)

## Abstract (arXiv)

While diffusion model fine-tuning offers a powerful approach for customizing pre-trained models to generate specific objects, it frequently suffers from overfitting when training samples are limited, compromising both generalization capability and output diversity. This paper tackles the challenging yet most impactful task of adapting a diffusion model using just a single concept image, as single-image customization holds the greatest practical potential. We introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework specifically designed for diffusion model personalization. We show that higher diffusion timesteps are more prone to overfitting than lower ones, necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates two key innovations: (1) a dynamic fine-tuning strategy that adjusts rank-constrained updates based on diffusion timesteps, and (2) a weight parametrization technique that ensures independence between adapter components through orthogonal initialization. Extensive experiments on SD-XL and FLUX-1.dev show that T-LoRA and its individual components outperform standard LoRA and other diffusion model personalization techniques, achieving a superior balance between concept fidelity and text alignment. Project page is available at https://controlgenai.github.io/T-LoRA/.