# TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models

arXiv: https://arxiv.org/abs/2508.08812 · HTML: https://mlsanity.com/paper/2508.08812/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs](https://mlsanity.com/paper/2311.13600/) (2023)
- [Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning](https://mlsanity.com/paper/2307.11410/) (2023)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2306.07967/) (2023)
- [ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models](https://mlsanity.com/paper/2305.16225/) (2023)
- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs](https://mlsanity.com/paper/2502.18461/) (2025)
- [Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigm](https://mlsanity.com/paper/2403.11781/) (2024)
- [Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models](https://mlsanity.com/paper/2404.04243/) (2024)
- [An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesis](https://mlsanity.com/paper/2311.11919/) (2023)
- [LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation](https://mlsanity.com/paper/2504.07448/) (2025)
- [Ada-adapter:Fast Few-shot Style Personlization of Diffusion Model with Pre-trained Image Encoder](https://mlsanity.com/paper/2407.05552/) (2024)
- [GLAD: Generalizable Tuning for Vision-Language Models](https://mlsanity.com/paper/2507.13089/) (2025)

## Built on by (most cited first, 5 of 5)

- [LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation](https://mlsanity.com/paper/2603.10899/) (2026)
- [ControlLight: Towards Controllable, Consistent, and Generalizable Low-Light Enhancement](https://mlsanity.com/paper/2605.25569/) (2026)
- [Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting](https://mlsanity.com/paper/2606.03792/) (2026)
- [V-Engram: Trigger-Indexed External Memory for Modular Text-to-Image Personalization](https://mlsanity.com/paper/2609.37198/) (2026)
- [CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork](https://mlsanity.com/paper/2610.01331/) (2026)

## Abstract (arXiv)

Personalized text-to-image generation aims to synthesize novel images of a specific subject or style using only a few reference images. Recent methods based on Low-Rank Adaptation (LoRA) enable efficient single-concept customization by injecting lightweight, concept-specific adapters into pre-trained diffusion models. However, combining multiple LoRA modules for multi-concept generation often leads to identity missing and visual feature leakage. In this work, we identify two key issues behind these failures: (1) token-wise interference among different LoRA modules, and (2) spatial misalignment between the attention map of a rare token and its corresponding concept-specific region. To address these issues, we propose Token-Aware LoRA (TARA), which introduces a token mask to explicitly constrain each module to focus on its associated rare token to avoid interference, and a training objective that encourages the spatial attention of a rare token to align with its concept region. Our method enables training-free multi-concept composition by directly injecting multiple independently trained TARA modules at inference time. Experimental results demonstrate that TARA enables efficient multi-concept inference and effectively preserving the visual identity of each concept by avoiding mutual interference between LoRA modules. The code and models are available at https://github.com/YuqiPeng77/TARA.