mlsanityData to 14 Oct 2026

MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition

arXiv 2411.01016 ↗First posted 1 Nov 2024cs.LG cs.AIcited by 57

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with considerably fewer activated parameters. Despite this efficiency, their enormous parameter size still leads to high deployment costs. In this paper, we introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. First, in the inter-expert pruning stage, we analyze the importance of each layer and propose the Layer-wise Genetic Search and Block-wise KT-Reception Field with the non-uniform pruning ratio to prune the individual expert. Second, in the intra-expert decomposition stage, we apply the low-rank decomposition to further compress the parameters within the remaining experts. Extensive experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8×7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks. The code will be available at urlhttps://github.com/xiaochengsky/MoEI-2.git

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 28 of its 42 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

28 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques4 Jun
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  3. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  4. 2024Mixtral of Experts8 Jan
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
9 more
  1. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  2. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  3. 2022Language model compression with weighted low-rank factorizationcited by 195
  4. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  5. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  7. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  8. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  9. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 20; 2026: 35.

57 papers.

Sort:

By area:

  1. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  3. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  4. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  5. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 26
  6. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  7. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  8. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  9. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  10. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  11. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  12. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
38 more
  1. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  2. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  3. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  4. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 9
  5. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  6. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  7. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 7
  8. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Modelscited by 6
  9. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inferencecited by 6
  10. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  11. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  12. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inferencecited by 4
  13. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  14. 2026Path-Constrained Mixture-of-Expertscited by 4
  15. 2025HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Spacecited by 3
  16. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Modelscited by 3
  17. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacingcited by 2
  18. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  19. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsitycited by 1
  20. 2026Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspectivecited by 1
  21. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  22. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 1
  23. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  24. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  25. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  26. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1
  27. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  28. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  29. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocationcited by 1
  30. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  31. 2025InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generationcited by 0
  32. 2026ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbitscited by 0
  33. 2026Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlapcited by 0
  34. 2026BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantizationcited by 0
  35. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  36. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0
  37. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  38. 2026EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decodingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  2. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  3. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  4. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
  5. 2026PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
  6. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  7. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  8. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts