# Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation

arXiv: https://arxiv.org/abs/2403.07500 · HTML: https://mlsanity.com/paper/2403.07500/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2206.06522/) (2022)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [StyleDrop: Text-to-Image Generation in Any Style](https://mlsanity.com/paper/2306.00983/) (2023)
- [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](https://mlsanity.com/paper/2212.03145/) (2022)
- [KronA: Parameter Efficient Tuning with Kronecker Adapter](https://mlsanity.com/paper/2212.10650/) (2022)
- [Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation](https://mlsanity.com/paper/2309.14859/) (2023)
- [One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2306.07967/) (2023)
- [Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing](https://mlsanity.com/paper/2309.15664/) (2023)
- [Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation](https://mlsanity.com/paper/2104.08771/) (2021)
- [Pufferfish: Communication-efficient Models At No Extra Cost](https://mlsanity.com/paper/2103.03936/) (2021)
- [Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works](https://mlsanity.com/paper/2210.08477/) (2022)

## Built on by (most cited first, 8 of 8)

- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [Unified Parameter-Efficient Unlearning for LLMs](https://mlsanity.com/paper/2412.00383/) (2024)
- [ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation](https://mlsanity.com/paper/2502.09411/) (2025)
- [Text-to-Image Synthesis: A Decade Survey](https://mlsanity.com/paper/2411.16164/) (2024)
- [A New Chinese Landscape Paintings Generation Model based on Stable Diffusion using DreamBooth](https://mlsanity.com/paper/2408.08561/) (2024)
- [FaR: Enhancing Multi-Concept Text-to-Image Diffusion via Concept Fusion and Localized Refinement](https://mlsanity.com/paper/2504.03292/) (2025)
- [HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models](https://mlsanity.com/paper/2609.31154/) (2026)

## Abstract (arXiv)

The objective of personalization and stylization in text-to-image is to instruct a pre-trained diffusion model to analyze new concepts introduced by users and incorporate them into expected styles. Recently, parameter-efficient fine-tuning (PEFT) approaches have been widely adopted to address this task and have greatly propelled the development of this field. Despite their popularity, existing efficient fine-tuning methods still struggle to achieve effective personalization and stylization in T2I generation. To address this issue, we propose block-wise Low-Rank Adaptation (LoRA) to perform fine-grained fine-tuning for different blocks of SD, which can generate images faithful to input prompts and target identity and also with desired style. Extensive experiments demonstrate the effectiveness of the proposed method.