mlsanityData to 14 Oct 2026

EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

arXiv 2607.01789 ↗First posted 2 Jul 2026cs.LG cs.AIcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 18 of its 26 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

18 of 26 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model12 Nov
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 15

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
3 more
  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  3. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMscited by 0
  2. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
  2. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
  3. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
  4. 2025FURINA: Free from Unmergeable Router via LINear Aggregation of mixed experts
  5. 2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
  6. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
  7. 2026Routing-Free Mixture-of-Experts
  8. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation