mlsanityData to 7 Oct 2026

REAM: Merging Improves Pruning of Experts in LLMs

arXiv 2604.04356 ↗First posted 6 Apr 2026cs.AI cs.CL cs.LGcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 48 of its 58 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

48 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density10 Feb
  2. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Merging16 Oct
  3. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compression15 Oct
  4. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging29 Jun
  5. 2025Qwen3 Technical Report14 May
  6. 2025Delta Decompression for MoE-based LLMs Compression24 Feb
  7. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition1 Nov
  8. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  9. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  10. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  11. 2024Mixtral of Experts8 Jan
  12. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Experts15 Oct
2 more
  1. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer15 Oct
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
22 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  5. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  7. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  8. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  9. 2025Qwen2.5-1M Technical Reportcited by 158
  10. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  11. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  13. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  14. 2026Qwen3-Coder-Next Technical Reportcited by 60
  15. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  16. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  17. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  18. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
  19. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  20. 2025On Linear Mode Connectivity of Mixture-of-Experts Architecturescited by 8
  21. 2025CAMEx: Curvature-aware Merging of Expertscited by 4
  22. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  2. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Accelerationcited by 2
  3. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  4. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  5. 2026Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Modelscited by 1
  6. 2026Reassessing Muon for Matrix Factorizationcited by 0
  7. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMscited by 0
  8. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
  9. 2026DIET: Deletion-response Expert Trimming for Video Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  2. 2025Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
  3. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
  4. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  5. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  6. 2026Higher-order pruning of experts in mixture-of-experts language models
  7. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  8. 2026LLMs can Compress LLMs: Adaptive Pruning by Agents