Data to 7 Oct 2026

Higher Layers Need More LoRA Experts

arXiv 2402.08562 ↗First posted 13 Feb 2024cs.CL cs.AIcited by 101

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this statement also apply to parameter-efficient MoE? In this paper, we introduce a novel parameter-efficient MoE method, textitMoE-LoRA with Layer-wise Expert Allocation (MoLA) for Transformer-based models, where each model layer has the flexibility to employ a varying number of LoRA experts. We investigate several architectures with varying layer-wise expert configurations. Experiments on six well-known NLP and commonsense QA benchmarks demonstrate that MoLA achieves equal or superior performance compared to all baselines. We find that allocating more LoRA experts to higher layers further enhances the effectiveness of models with a certain number of experts in total. With much fewer parameters, this allocation strategy outperforms the setting with the same number of experts in every layer. This work can be widely used as a plug-and-play parameter-efficient tuning approach for various applications. The code is available at https://github.com/GCYZSL/MoLA.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 28 of its 38 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

28 of 38 references.

Closest ancestors · 1

Same direction, newest first

2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin15 Dec

Other references · 27

2017Attention Is All You Needcited by 59,932
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881

15 more

2022Large Language Models Encode Clinical Knowledgecited by 1,807
2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
2024Mixtral of Expertscited by 1,785
2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
2023What Do Llamas Really Think? Revealing Preference Biases in Language Model Representationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 38; 2026: 29.

101 papers.

Sort:

By area:

2024A Survey on Mixture of Experts in Large Language Modelscited by 233
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
2024A Survey on LoRA of Large Language Modelscited by 67
2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
2024On Large Language Model Continual Unlearningcited by 51
2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 41
2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33

38 more

2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 18
2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
2024Low-Rank Interconnected Adaptation across Layerscited by 10
2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 10
2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
2025Ensembles of Low-Rank Expert Adapterscited by 9
2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
2024MEMoE: Enhancing Model Editing with Mixture of Experts Adaptorscited by 8
2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learningcited by 7
2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modulescited by 7
2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
2025Learning without Isolation: Pathway Protection for Continual Learningcited by 6
2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectorscited by 6
2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
2024Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devicescited by 5
2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Expertscited by 5
2025CoLA: Collaborative Low-Rank Adaptationcited by 5
2025Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognitioncited by 5
2025Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptationcited by 5
2025LoRA-Gen: Specializing Large Language Model via Online LoRA Generationcited by 5
2025ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needscited by 5
2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 5

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way