MoEITS: A Green AI approach for simplifying MoE-LLMs
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
Large language models are transforming all areas of academia and industry, attracting the attention of researchers, professionals, and the general public. In the trek for more powerful architectures, Mixture-of-Experts, inspired by ensemble models, have emerged as one of the most effective ways to follow. However, this implies a high computational burden for both training and inference. To reduce the impact on computing and memory footprint as well as the energy consumption, simplification methods has arisen as very effective procedures. In this paper, an original algorithm, MoEITS, for MoE-LLMs simplification is presented. The algorithm is characterized by a refined simplicity, underpinned by standardized Information Theoretic frameworks. MoEITS is analyzed in depth from theoretical and practical points of view. Its computational complexity is studied. Its performance on the accuracy of the simplified LLMs and the reduction rate achieved is assessed through a thoroughly designed experimentation. This empirical evaluation includes a comparison with state-of-the-art MoE-LLM pruning methods applied on Mixtral 8×7B, Qwen1.5-2.7B, and DeepSeek-V2-Lite. The extensive experimentation conducted demonstrates that MoEITS outperforms state-of-the-art techniques by generating models that are both effective across all benchmarks and computationally efficient. The code implementing the method will be available at https://github.com/luisbalru/MoEITS.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 58 references that are in the map. First posted on arXiv 12 Apr 2026.
Builds on
34 of 58 references.
Closest ancestors · 12
Same direction, newest first
- 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models25 Nov
- 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs12 Sep
- 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models3 Aug
- 2025DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts11 Jun
- 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning25 Jan
- 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition1 Nov
- 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
- 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More8 Oct
- 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning10 Sep
- 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
- 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan
Other references · 22
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
- 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
10 more
- 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
- 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
- 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
- 2023A Systematic Review of Green AIcited by 77
- 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
- 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
- 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 25
- 2023Emotion Detection for Misinformation: A Reviewcited by 22
- 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
- 2025Carbon Footprint Evaluation of Code Generation through LLM as a Servicecited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024A Survey on Mixture of Experts in Large Language Models
- 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
- 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
- 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
- 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
- 2025Faster MoE LLM Inference for Extremely Large Models
- 2025Mixture of Experts in Large Language Models
- 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models