mlsanityData to 14 Oct 2026

ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration

arXiv 2503.06881 ↗First posted 10 Mar 2025cs.LGcited by 12

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) Transformer, the backbone architecture of multiple phenomenal language models, leverages sparsity by activating only a fraction of model parameters for each input token. The sparse structure, while allowing constant time costs, results in space inefficiency: we still need to load all the model parameters during inference. We introduce ResMoE, an innovative MoE approximation framework that utilizes Wasserstein barycenter to extract a common expert (barycenter expert) and approximate the residuals between this barycenter expert and the original ones. ResMoE enhances the space efficiency for inference of large-scale MoE Transformers in a one-shot and data-agnostic manner without retraining while maintaining minimal accuracy loss, thereby paving the way for broader accessibility to large language models. We demonstrate the effectiveness of ResMoE through extensive experiments on Switch Transformer, Mixtral, and DeepSeekMoE models. The results show that ResMoE can reduce the number of parameters in an expert by up to 75% while maintaining comparable performance. The code is available at https://github.com/iDEA-iSAIL-Lab-UIUC/ResMoE.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 71 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

29 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2024Mixtral of Experts8 Jan
  4. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget29 Aug
  5. 2023MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models18 Jul
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Mistral 7Bcited by 3,164
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2018Computational Optimal Transportcited by 780
  12. 2018Rethinking the Value of Network Pruningcited by 599
11 more
  1. 2020What is the State of Neural Network Pruning?cited by 583
  2. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  3. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  4. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  5. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  6. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  7. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  8. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  9. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 59
  10. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  11. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025PLANETALIGN: A Comprehensive Python Library for Benchmarking Network Alignmentcited by 10
  2. 2025Hierarchical LoRA MoE for Efficient CTR Model Scalingcited by 10
  3. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  4. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 10
  5. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  6. 2026Subspace Alignment for Vision-Language Model Test-time Adaptationcited by 5
  7. 2026AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMscited by 4
  8. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Modelscited by 3
  9. 2026Harnessing Generalist Agents for Contextualized Time Seriescited by 3
  10. 2025Embedding Empirical Distributions for Computing Optimal Transport Mapscited by 0
  11. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  12. 2026Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  2. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
  3. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  4. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
  5. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
  6. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
  7. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  8. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection