mlsanityData to 7 Oct 2026

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

arXiv 2403.03432 ↗First posted 6 Mar 2024cs.CL cs.AIcited by 38

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models (LLMs). However, achieving the right balance of data is crucial to prevent catastrophic forgetting and interference between tasks. To address these limitations and enhance training flexibility, we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and parameter-efficient tuning method designed for multi-task learning with LLMs. In this paper, we start by individually training multiple domain-specific LoRA modules using corresponding supervised corpus data. These LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE). Subsequently, we combine the multiple LoRAs using an explicit routing strategy and introduce domain labels to facilitate multi-task learning, which help prevent interference between tasks and ultimately enhances the performance of each individual task. Furthermore, each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation. Experiments on diverse tasks demonstrate superior and robust performance, which can further promote the wide application of domain-specific LLMs.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 15 of its 26 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

15 of 26 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  10. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  12. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
3 more
  1. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  2. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  3. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 119

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 15; 2026: 14.

38 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  3. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  4. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  5. 2024A Survey on LoRA of Large Language Modelscited by 67
  6. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  7. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  8. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  9. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  10. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  11. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12
  12. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
26 more
  1. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  2. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  3. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  4. 2025Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuningcited by 6
  5. 2025Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptationcited by 5
  6. 2025TokMem: One-Token Procedural Memory for Large Language Modelscited by 5
  7. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  8. 2025A Survey of LLM × DATAcited by 4
  9. 2025HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalizationcited by 4
  10. 2026Spanning the Visual Analogy Space with a Weight Basis of LoRAscited by 4
  11. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3
  12. 2024VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognitioncited by 2
  13. 2025Communication-Efficient and Personalized Federated Foundation Model Fine-Tuning via Tri-Matrix Adaptationcited by 2
  14. 2025Low-Rank Expert Merging for Multi-Source Domain Adaptation in Person Re-Identificationcited by 2
  15. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  16. 2026MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Expertscited by 2
  17. 2025NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptationcited by 1
  18. 2025FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCAcited by 1
  19. 2026Efficient Handwriting-Based Alzheimer,s Disease Diagnosis Using a Low-Rank Mixture of Experts Deep Learning Frameworkcited by 0
  20. 2026Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routingcited by 0
  21. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMscited by 0
  22. 2026MED-DSLC: Multi-Expert-Domain Classification via Domain Supervision and Logit Calibrationcited by 0
  23. 2026Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluationcited by 0
  24. 2026SE-MoLoRA: Shared-Expert LoRA Adapters for Domain-Specific Photographic Assessmentcited by 0
  25. 2026PoEM: Predicting RL Outcomes from Existing Policiescited by 0
  26. 2026Learning to Predict Distributions over Weight Updates for Test-Time Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
  2. 2025From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
  3. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
  4. 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
  5. 2025D2LoRA: Data-Driven LoRA Initialization for Low Resource Tasks
  6. 2024T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning
  7. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  8. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs