mlsanityData to 14 Oct 2026

Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

arXiv 2506.13479 ↗First posted 16 Jun 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

35 of 46 references.

References · 35

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
23 more
  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  2. 2020What is being transferred in transfer learning?cited by 368
  3. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  4. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  5. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  6. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  7. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  8. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  9. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  10. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  11. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  12. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
  13. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  14. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  15. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  16. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  17. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28
  18. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 23
  19. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  20. 2021The Separation Capacity of Random Neural Networkscited by 14
  21. 2024Realistic Evaluation of Model Merging for Compositional Generalizationcited by 12
  22. 2021AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapterscited by 8
  23. 2022Randomly Initialized One-Layer Neural Networks Make Data Linearly Separablecited by 6

Built on by

Not cited yet.