mlsanityData to 14 Oct 2026

DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models

arXiv 2503.01359 ↗First posted 3 Mar 2025cs.LGcited by 10

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Upcycled Mixture-of-Experts (MoE) models have shown great potential in various tasks by converting the original Feed-Forward Network (FFN) layers in pre-trained dense models into MoE layers. However, these models still suffer from significant parameter inefficiency due to the introduction of multiple experts. In this work, we propose a novel DeRS (Decompose, Replace, and Synthesis) paradigm to overcome this shortcoming, which is motivated by our observations about the unique redundancy mechanisms of upcycled MoE experts. Specifically, DeRS decomposes the experts into one expert-shared base weight and multiple expert-specific delta weights, and subsequently represents these delta weights in lightweight forms. Our proposed DeRS paradigm can be applied to enhance parameter efficiency in two different scenarios, including: 1) DeRS Compression for inference stage, using sparsification or quantization to compress vanilla upcycled MoE models; and 2) DeRS Upcycling for training stage, employing lightweight sparse or low-rank matrixes to efficiently upcycle dense models into MoE models. Extensive experiments across three different tasks show that the proposed methods can achieve extreme parameter efficiency while maintaining the performance for both training and compression of upcycled MoE models.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 47 of its 53 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

47 of 53 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
35 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  13. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  14. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  15. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  16. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  17. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  18. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  19. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  20. 2023From Sparse to Soft Mixtures of Expertscited by 248
  21. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  22. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  23. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  24. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  25. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  26. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  27. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  28. 2024Stable LM 2 1.6B Technical Reportcited by 90
  29. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  30. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 59
  31. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  32. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
  33. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  34. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 46
  35. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Advancing Expert Specialization for Better MoEcited by 39
  2. 2025MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Modelscited by 4
  3. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4
  4. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  5. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilitiescited by 3
  6. 2025CTTS: Collective Test-Time Scalingcited by 3
  7. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1
  8. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  9. 2026Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentationcited by 0
  10. 2026Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projectionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  2. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts
  3. 2023Revisiting Single-gated Mixtures of Experts
  4. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  5. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  7. 2025Sigma-MoE-Tiny Technical Report
  8. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization