mlsanityData to 7 Oct 2026

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules

arXiv 2508.02587 ↗First posted 4 Aug 2025cs.LG cs.AI cs.CLcited by 7

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) benefits from a dynamic routing mechanism among their specialized experts, which existing Parameter- Efficient Fine-Tuning (PEFT) strategies fail to leverage. This motivates us to investigate whether adaptation modules themselves should incorporate routing mechanisms to align with MoE's multi-expert architecture. We analyze dynamics of core components when applying PEFT to MoE language models and examine how different routing strategies affect adaptation effectiveness. Extensive experiments adapting OLMoE-1B-7B and Mixtral-8x7B on various commonsense and math reasoning tasks validate the performance and efficiency of our routed approach. We identify the optimal configurations for different scenarios and provide empirical analyses with practical insights to facilitate better PEFT and MoE applications.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 38 of its 59 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

38 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2020GLU Variants Improve Transformercited by 1,724
24 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2016Language Modeling with Gated Convolutional Networkscited by 688
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  9. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  11. 2016Solving General Arithmetic Word Problemscited by 353
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  13. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  14. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  15. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  16. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  17. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  18. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  19. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  20. 2024Higher Layers Need More LoRA Expertscited by 101
  21. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  22. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  23. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  24. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adaptercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Routing-Free Mixture-of-Expertscited by 2
  2. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptationcited by 1
  3. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  4. 2026Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Lawscited by 0
  5. 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptationcited by 0
  6. 2026ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMscited by 0
  7. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
  2. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models
  3. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  4. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  5. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
  6. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  7. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  8. 2025Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models