mlsanityData to 7 Oct 2026

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

arXiv 2401.02731 ↗First posted 5 Jan 2024cs.AIcited by 22

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5. Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 48 of its 72 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

48 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models24 May
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
31 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  13. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  14. 2022Emergent Abilities of Large Language Modelscited by 2,483
  15. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  17. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  18. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  19. 2017A Survey on Multi-Task Learningcited by 838
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  22. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  23. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  24. 2023From Sparse to Soft Mixtures of Expertscited by 245
  25. 2019Universal Approximation with Deep Narrow Networkscited by 198
  26. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  27. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  28. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  29. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  30. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  31. 2021Speeding up Deep Model Training by Sharing Weights and Then Unsharingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  2. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  3. 2024Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Modelscited by 25
  4. 2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
  5. 2025Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generationcited by 16
  6. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
  7. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
  8. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertscited by 7
  9. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
  10. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  11. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
  12. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
10 more
  1. 2025Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuningcited by 3
  2. 2024DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Modelscited by 2
  3. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuningcited by 2
  4. 2025Ghost in the Transformer: Detecting Model Reuse with Invariant Spectral Signaturescited by 2
  5. 2026AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimizationcited by 2
  6. 2026MoRSE: Task-Oriented Multi-Agent System with Mixture of Role-Subtask Expertscited by 2
  7. 2025Efficient OpAmp Adaptation for Zoom Attention to Golden Contextscited by 1
  8. 2026Mixture of Masters: Sparse Chess Language Models with Player Routingcited by 1
  9. 2026Sparse Spectral LoRA: Routed Experts for Medical VLMscited by 1
  10. 2026LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  2. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
  3. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  4. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  6. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
  7. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  8. 2026Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution