mlsanityData to 14 Oct 2026

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation

arXiv 2509.16882 ↗First posted 21 Sep 2025cs.LG cs.AI cs.CLcited by 10

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Mixture-of-Experts (MoE) models offer immense capacity via sparsely gated expert subnetworks, yet adapting them to multiple domains without catastrophic forgetting remains an open challenge. Existing approaches either incur prohibitive computation, suffer cross-domain interference, or require separate runs per domain. We propose DES-MoE, a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models. DES-MoE addresses catastrophic forgetting through three innovations: (1) an adaptive router balancing pre-trained knowledge retention and task-specific updates via distillation, (2) real-time expert-domain correlation mapping to isolate domain-specific gradients, and (3) a three-phase adaptive fine-tuning schedule that progressively freezes non-specialized parameters. Evaluated on six domains (math, code, law, etc.), DES-MoE matches single-domain ESFT performance while training one unified model, reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6, and achieves 68% faster convergence than conventional methods. Our work establishes dynamic expert isolation as a scalable paradigm for multi-task MoE adaptation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 33 of its 49 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

33 of 49 references.

References · 33

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
21 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  7. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  8. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  9. 2021Hash Layers For Large Sparse Modelscited by 237
  10. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  11. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  12. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  13. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  14. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  15. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  16. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
  17. 2021Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translationcited by 74
  18. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  19. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  20. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
  21. 2020ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalizationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 8
  2. 2026Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Modelscited by 4
  3. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  4. 2026PathMark: Protecting Intellectual Property of Mixture-of-Expert LLMs via Path Watermarkscited by 4
  5. 2026Learning Transferable Topology Priors for Multi-Agent LLM Collaboration Across Domainscited by 2
  6. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMscited by 1
  7. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Expertscited by 0
  8. 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMscited by 0
  9. 2026SE-MoLoRA: Shared-Expert LoRA Adapters for Domain-Specific Photographic Assessmentcited by 0
  10. 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  2. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillation
  3. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  4. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  5. 2023Improving Expert Specialization in Mixture of Experts
  6. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  7. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  8. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation