# Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

arXiv: https://arxiv.org/abs/2609.22237 · HTML: https://mlsanity.com/paper/2609.22237/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [Parameter Competition Balancing for Model Merging](https://mlsanity.com/paper/2410.02396/) (2024)
- [Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering](https://mlsanity.com/paper/2409.16167/) (2024)
- [Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging](https://mlsanity.com/paper/2505.15875/) (2025)
- [Merge before Forget: A Single LoRA Continual Learning via Continual Merging](https://mlsanity.com/paper/2512.23017/) (2025)
- [AdaRank: Adaptive Rank Pruning for Enhanced Model Merging](https://mlsanity.com/paper/2503.22178/) (2025)
- [ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation](https://mlsanity.com/paper/2603.02945/) (2026)
- [Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems](https://mlsanity.com/paper/2511.22880/) (2025)
- [Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter](https://mlsanity.com/paper/2606.03723/) (2026)

## Abstract (arXiv)

Merging low-rank adapters (LoRAs) promises to eliminate the overhead of swapping task-specific weights at inference time. However, existing merging methods assume every layer needs the same rank budget. Further, some methods assume that rank budget needs to be split equally among the tasks too. We show this uniform-budget assumption is a major source of the performance gap between merged and per-task LoRAs. However, rank selection is an NP hard problem. To this end, we introduce Net Utility, a data free metric that first decomposes every task LoRA by its Singular Value Decomposition (SVD) and scores each of those singular directions by its task utility and its interference with other tasks directions. Next, we globally pool these scores to select singular directions with the highest values with a constraint on the total number of directions selected. The proposed Net Utility metric is applied on top of five different merging methods across three different merging spaces. The merging is done over two sets of tasks, vision and language tasks. Net utility based rank allocation outperforms its counterparts without that allocation. On average, over vision tasks it achieves +2.1% improvement in performance, and +2.2% improvement over the language tasks.