mlsanityData to 7 Oct 2026

Towards Modular LLMs by Building and Reusing a Library of LoRAs

arXiv 2405.11157 ↗First posted 18 May 2024cs.LG cs.CLcited by 72

Not on the map. Why?

Abstract (arXiv)

The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for new tasks. We study how to best build a library of adapters given multi-task data and devise techniques for both zero-shot and supervised task generalization through routing in such library. We benchmark existing approaches to build this library and introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters, indirectly optimizing for transfer across the multi-task dataset. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters for new inputs without the need for retraining. We experiment with several LLMs, such as Phi-2 and Mistral, on a wide array of held-out tasks, verifying that MBC-based adapters and Arrow routing lead to superior generalization to new tasks. We make steps towards creating modular, adaptable LLMs that can match or outperform traditional joint training.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 81 of its 96 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

81 of 96 references.

References · 81

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Mistral 7Bcited by 3,065
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2021Learning to Prompt for Continual Learningcited by 938
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  13. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  15. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  16. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  17. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  18. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  20. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  21. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  22. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  23. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  24. 2023Towards Generalist Biomedical AIcited by 286
  25. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  26. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  27. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  28. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  29. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  30. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  31. 2024Mixture of LoRA Expertscited by 179
  32. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  33. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  34. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  35. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  36. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  37. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  38. 2020Exploring and Predicting Transferability across NLP Taskscited by 126

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 28; 2026: 28.

72 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  3. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  4. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  5. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  6. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  7. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  8. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  9. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 14
  10. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  11. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 11
  12. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
38 more
  1. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 8
  2. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Expertscited by 7
  3. 2025Permissioned LLMs: Enforcing Access Control in Large Language Modelscited by 7
  4. 2025LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Taskscited by 7
  5. 2025Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDPcited by 7
  6. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bitscited by 6
  7. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  8. 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
  9. 2026Effective LoRA Adapter Routing using Task Representationscited by 6
  10. 2024MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Expertscited by 4
  11. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  12. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  13. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 4
  14. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  15. 2025HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalizationcited by 4
  16. 2025Exploring How LLMs Capture and Represent Domain-Specific Knowledgecited by 3
  17. 2025One Rank at a Time: Cascading Error Dynamics in Sequential Learningcited by 3
  18. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  19. 2024DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Modelscited by 2
  20. 2024Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projectioncited by 2
  21. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 2
  22. 2026Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learningcited by 2
  23. 2026The Appeal and Reality of Recycling LoRAs with Adaptive Mergingcited by 2
  24. 2026Adapting Time Series Foundation Models through Data Mixturescited by 2
  25. 2026How LoRA Remembers? A Parametric Memory Law for LLM Finetuningcited by 2
  26. 2026Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memorycited by 2
  27. 2025Tensorized Clustered LoRA Merging for Multi-Task Interferencecited by 1
  28. 2025A Lightweight Framework for Trigger-Guided LoRA-Based Self-Adaptation in LLMscited by 1
  29. 2025SEQR: Secure and Efficient QR-based LoRA Routingcited by 1
  30. 2026Rethinking LLM Ensembling from the Perspective of Mixture Modelscited by 1
  31. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioningcited by 1
  32. 2024MoIN: Mixture of Introvert Experts to Upcycle an LLMcited by 0
  33. 2025GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtractioncited by 0
  34. 2025Two Is Better Than One: Rotations Scale LoRAscited by 0
  35. 2025Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectivenesscited by 0
  36. 2025Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Expertscited by 0
  37. 2025Advancing Compositional LLM Reasoning with Structured Task Relations in Interactive Multimodal Communicationscited by 0
  38. 2025Delta Activations: A Representation for Finetuned Large Language Modelscited by 0

50 most cited of 72 All 72 (JSON)