mlsanityData to 7 Oct 2026

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

arXiv 2307.13269 ↗First posted 25 Jul 2023cs.CL cs.AIcited by 359

Not on the map. Why?

Cited by 31 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem. Our code is available at https://github.com/sail-sg/lorahub, and all the pre-trained LoRA modules are released at https://huggingface.co/lorahub.

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

46 of 59 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Editing Models with Task Arithmeticcited by 1,076
34 more
  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  9. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  11. 2021MetaICL: Learning to Learn In Contextcited by 450
  12. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  13. 2023Adapting Language Models to Compress Contextscited by 350
  14. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  15. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  16. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  17. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  18. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  19. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  20. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  21. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  22. 2018Graph HyperNetworks for Neural Architecture Searchcited by 131
  23. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  24. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  25. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  26. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  27. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  28. 2023Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domaincited by 38
  29. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  30. 2020Versatile Black-Box Optimizationcited by 22
  31. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  32. 2022Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learningcited by 11
  33. 2022SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understandingcited by 10
  34. 2021Neural Network Module Decomposition and Recompositioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 114; 2025: 118; 2026: 104.

359 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  4. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  7. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  8. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  9. 2023Efficient Large Language Models: A Surveycited by 183
  10. 2024Mixture of LoRA Expertscited by 179
  11. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  12. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
38 more
  1. 2024Model merging with SVD to tie the Knotscited by 118
  2. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  3. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  4. 2024Knowledge Fusion of Large Language Modelscited by 110
  5. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  6. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  7. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  8. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  9. 2023Deep Model Fusion: A Surveycited by 89
  10. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  11. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  12. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  13. 2024Multi-LoRA Composition for Image Generationcited by 80
  14. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  15. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  16. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  17. 2024Parameter Competition Balancing for Model Mergingcited by 71
  18. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  19. 2024A Survey on LoRA of Large Language Modelscited by 67
  20. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  21. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 64
  22. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  23. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
  24. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  25. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  26. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  27. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  28. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  29. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  30. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  31. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  32. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 36
  33. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  34. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  35. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  36. 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operationcited by 33
  37. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
  38. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32

50 most cited of 359 All 359 (JSON)