mlsanityData to 14 Oct 2026

TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts

arXiv 2606.09885 ↗First posted 3 Jun 2026cs.LG stat.MLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, reserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 36 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

27 of 36 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning19 Sep
  2. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts11 Aug
  3. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning26 Nov
  4. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  5. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  6. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection27 Feb
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  8. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Experts15 Oct
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  9. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  10. 2024Higher Layers Need More LoRA Expertscited by 102
  11. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  12. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
5 more
  1. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  2. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  3. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 10
  4. 2024DA-MoE: Addressing Depth-Sensitivity in Graph-Level Analysis through Mixture of Expertscited by 7
  5. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
  2. 2026Sparsely gated tiny linear experts
  3. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  4. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoE
  5. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  6. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
  7. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
  8. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts