Data to 14 Oct 2026

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

arXiv 2404.15159 ↗First posted 22 Apr 2024cs.CL cs.AIcited by 147

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Fine-tuning Large Language Models (LLMs) is a common practice to adapt pre-trained models for specific applications. While methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multi-task scenarios. In contrast, Mixture-of-Expert (MoE) models, such as Mixtral 8x7B, demonstrate remarkable performance in multi-task learning scenarios while maintaining a reduced parameter count. However, the resource requirements of these MoEs remain challenging, particularly for consumer-grade GPUs with less than 24GB memory. To tackle these challenges, we propose MixLoRA, an approach to construct a resource-efficient sparse MoE model based on LoRA. MixLoRA inserts multiple LoRA-based experts within the feed-forward network block of a frozen pre-trained dense model and employs a commonly used top-k router. Unlike other LoRA-based MoE methods, MixLoRA enhances model performance by utilizing independent attention-layer LoRA adapters. Additionally, an auxiliary load balance loss is employed to address the imbalance problem of the router. Our evaluations show that MixLoRA improves about 9% accuracy compared to state-of-the-art PEFT methods in multi-task learning scenarios. We also propose a new high-throughput framework to alleviate the computation and memory bottlenecks during the training and inference of MOE models. This framework reduces GPU memory consumption by 40% and token computation latency by 30% during both training and inference.

Key facts. Cited by 147 papers in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 60 of its 69 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

60 of 69 references.

Closest ancestors · 5

Same direction, newest first

Other references · 55

2017Attention Is All You Needcited by 60,361
2020Language Models are Few-Shot Learnerscited by 32,419
2023GPT-4 Technical Reportcited by 18,396
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915

38 more

2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
2022Scaling Instruction-Finetuned Language Modelscited by 3,024
2022Training Compute-Optimal Large Language Modelscited by 3,019
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
2022Emergent Abilities of Large Language Modelscited by 2,512
2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
2024Mixtral of Expertscited by 1,823
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
2019Learning Deep Transformer Models for Machine Translationcited by 309
2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
2018Adaptive Input Representations for Neural Language Modelingcited by 226
2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 50; 2026: 65.

147 papers.

Sort:

By area:

2024A Survey on Mixture of Experts in Large Language Modelscited by 237
2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 25
2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 20
2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 18

38 more

2026Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuningcited by 17
2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 16
2024Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Expertscited by 15
2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
2025Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Expertscited by 14
2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 13
2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 13
2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 11
2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
2025Ensembles of Low-Rank Expert Adapterscited by 9
2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
2025TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Visioncited by 9
2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 9
2025MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learningcited by 9
2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
2024DataLab: A Unified Platform for LLM-Powered Business Intelligencecited by 8
2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modulescited by 8
2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learningcited by 7
2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
2024MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Expertscited by 6
2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 6
2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
2024FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Modelscited by 5
2025Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognitioncited by 5
2025LoRA-Gen: Specializing Large Language Model via Online LoRA Generationcited by 5
2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 5
2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 5

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way