mlsanityData to 7 Oct 2026

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

arXiv 2407.01906 ↗First posted 2 Jul 2024cs.CL cs.AI cs.LGcited by 21

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) is crucial for customizing Large Language Models (LLMs) with constrained resources. Although there have been various PEFT methods for dense-architecture LLMs, PEFT for sparse-architecture LLMs is still underexplored. In this work, we study the PEFT method for LLMs with the Mixture-of-Experts (MoE) architecture and the contents of this work are mainly threefold: (1) We investigate the dispersion degree of the activated experts in customized tasks, and found that the routing distribution for a specific task tends to be highly concentrated, while the distribution of activated experts varies significantly across different tasks. (2) We propose Expert-Specialized Fine-Tuning, or ESFT, which tunes the experts most relevant to downstream tasks while freezing the other experts and modules; experimental results demonstrate that our method not only improves the tuning efficiency, but also matches or even surpasses the performance of full-parameter fine-tuning. (3) We further analyze the impact of the MoE architecture on expert-specialized fine-tuning. We find that MoE models with finer-grained experts are more advantageous in selecting the combination of experts that are most relevant to downstream tasks, thereby enhancing both the training efficiency and effectiveness. Our code is available at https://github.com/deepseek-ai/ESFT.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 37 of its 46 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

37 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  2. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 33

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
21 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  6. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  10. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  11. 2021Hash Layers For Large Sparse Modelscited by 236
  12. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  13. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  14. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  15. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  16. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  17. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  18. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  19. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
  20. 2023Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuningcited by 29
  21. 2020ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalizationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 14.

21 papers.

Sort:

By area:

  1. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  2. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  3. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
  4. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  5. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  6. 2025Route Experts by Sequence, not by Tokencited by 5
  7. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  8. 2026DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Modelscited by 4
  9. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  10. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacingcited by 2
  11. 2026EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuningcited by 2
  12. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuningcited by 1
9 more
  1. 2026Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuningcited by 1
  2. 2026Modular Pretraining Enables Access Controlcited by 1
  3. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMscited by 1
  4. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  5. 2026Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuningcited by 0
  6. 2026LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Modelscited by 0
  7. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 0
  8. 2026RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Modelscited by 0
  9. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  3. 2025Advancing Expert Specialization for Better MoE
  4. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  5. 2025Mixture of Experts in Large Language Models
  6. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models
  7. 2025ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale
  8. 2024A Closer Look into Mixture-of-Experts in Large Language Models