mlsanityData to 14 Oct 2026

AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning

arXiv 2603.18492 ↗First posted 19 Mar 2026cs.LGcited by 5

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead. Existing task-agnostic expert-pruning methods are typically calibration-dependent: they estimate expert importance from routing or activation statistics on a calibration set, making pruning decisions sensitive to calibration-data variation while introducing substantial preprocessing cost. We propose AIMER (Absolute mean over root mean square IMportance for Expert Ranking), a simple calibration-free criterion that identifies more distinct experts by capturing the concentration pattern of expert weights, making it well suited for task-agnostic expert pruning. Across 7B to 47B MoE language models with distinct architectures and 16 diverse benchmarks, AIMER consistently delivers stronger capability balance across diverse tasks than existing calibration-free methods. Surprisingly, AIMER also achieves better balance than strong calibration-based expert-pruning baselines calibrated on the widely used task-agnostic C4 corpus, while requiring only 0.22--2.06 seconds to score all experts.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 40 of its 53 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

40 of 53 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoE6 Mar
  2. 2025MoE Pathfinder: Trajectory-driven Expert Pruning20 Dec
  3. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compression15 Oct
  4. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs12 Sep
  5. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering11 Oct
  6. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning10 Sep
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  8. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  10. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Experts15 Oct
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
17 more
  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  4. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  6. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  7. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  8. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  9. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  10. 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
  11. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  12. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  13. 2019Data-Independent Neural Pruning via Coresetscited by 35
  14. 2024Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruningcited by 20
  15. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  16. 2021RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Mergingcited by 19
  17. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  2. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  3. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  4. 2026ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMscited by 0
  5. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Higher-order pruning of experts in mixture-of-experts language models
  2. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
  3. 2026It Takes a MAESTRO To Prune Bad Experts
  4. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  6. 2026Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning
  7. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  8. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models