mlsanityData to 14 Oct 2026

PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model

arXiv 2411.08212 ↗First posted 12 Nov 2024cs.LG cs.AIcited by 9

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The Mixture-of-Experts (MoE) paradigm has emerged as a powerful approach for scaling transformers with improved resource utilization. However, efficiently fine-tuning MoE models remains largely underexplored. Inspired by recent works on Parameter-Efficient Fine-Tuning (PEFT), we present a unified framework for integrating PEFT modules directly into the MoE mechanism. Aligning with the core principles and architecture of MoE, our framework encompasses a set of design dimensions including various functional and composition strategies. By combining design choices within our framework, we introduce Parameter-Efficient Routed Fine-Tuning (PERFT) as a flexible and scalable family of PEFT strategies tailored for MoE models. Extensive experiments on adapting OLMoE-1B-7B and Mixtral-8×7B for commonsense and arithmetic reasoning tasks demonstrate the effectiveness, scalability, and intriguing dynamics of PERFT. Additionally, we provide empirical findings for each specific design choice to facilitate better application of MoE and PEFT.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 28 of its 65 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

28 of 65 references.

References · 28

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2024Mixtral of Expertscited by 1,823
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  12. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
16 more
  1. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  3. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  4. 2024Mixture of LoRA Expertscited by 181
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  6. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  7. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
  8. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  9. 2024Higher Layers Need More LoRA Expertscited by 102
  10. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  11. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  12. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 56
  13. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  14. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  15. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  16. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adaptercited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 3.

9 papers.

  1. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 51
  2. 2025CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Processcited by 51
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  4. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Expertscited by 6
  5. 2026DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Modelscited by 5
  6. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  7. 2026EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuningcited by 2
  8. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  9. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
  2. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models
  3. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
  4. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  5. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
  6. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
  7. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  8. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning