# LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation

arXiv: https://arxiv.org/abs/2412.05148 · HTML: https://mlsanity.com/paper/2412.05148/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 60 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 30 of 50)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2307.06949/) (2023)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023)
- [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023)
- [ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs](https://mlsanity.com/paper/2311.13600/) (2023)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning](https://mlsanity.com/paper/2307.11410/) (2023)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [LLaVA-Critic: Learning to Evaluate Multimodal Models](https://mlsanity.com/paper/2410.02712/) (2024)

## Built on by (most cited first, 19 of 19)

- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging](https://mlsanity.com/paper/2507.17706/) (2025)
- [Efficient Compositional Multi-tasking for On-device Large Language Models](https://mlsanity.com/paper/2507.16083/) (2025)
- [K-Merge: Online Continual Merging of Adapters for On-device Large Language Models](https://mlsanity.com/paper/2510.13537/) (2025)
- [DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generation](https://mlsanity.com/paper/2603.20470/) (2026)
- [Low-Rank Adaptation Redux for Large Models](https://mlsanity.com/paper/2604.21905/) (2026)
- [On-device System of Compositional Multi-tasking in Large Language Models](https://mlsanity.com/paper/2510.13848/) (2025)
- [Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation](https://mlsanity.com/paper/2603.21884/) (2026)
- [The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptation](https://mlsanity.com/paper/2604.23750/) (2026)
- [LoRA-Key: User-Centric LoRA Watermarking for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2605.29569/) (2026)
- [SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models](https://mlsanity.com/paper/2606.10617/) (2026)
- [DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation](https://mlsanity.com/paper/2606.29961/) (2026)
- [Data-driven Clustering and Merging of Adapters for On-device Large Language Models](https://mlsanity.com/paper/2601.17441/) (2026)
- [Training-Free Multi-Concept Image Editing](https://mlsanity.com/paper/2602.20839/) (2026)
- [Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting](https://mlsanity.com/paper/2606.03792/) (2026)
- [Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach](https://mlsanity.com/paper/2607.15661/) (2026)
- [DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation](https://mlsanity.com/paper/2607.17467/) (2026)
- [SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge](https://mlsanity.com/paper/2607.27497/) (2026)

## Abstract (arXiv)

Recent advancements in image generation models have enabled personalized image creation with both user-defined subjects (content) and styles. Prior works achieved personalization by merging corresponding low-rank adapters (LoRAs) through optimization-based methods, which are computationally demanding and unsuitable for real-time use on resource-constrained devices like smartphones. To address this, we introduce LoRA.rar, a method that not only improves image quality but also achieves a remarkable speedup of over 4000× in the merging process. We collect a dataset of style and subject LoRAs and pre-train a hypernetwork on a diverse set of content-style LoRA pairs, learning an efficient merging strategy that generalizes to new, unseen content-style pairs, enabling fast, high-quality personalization. Moreover, we identify limitations in existing evaluation metrics for content-style quality and propose a new protocol using multimodal large language models (MLLMs) for more accurate assessment. Our method significantly outperforms the current state of the art in both content and style fidelity, as validated by MLLM assessments and human evaluations.