Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning
On the map: Mixture-of-experts → MoE theory and gating functions also fits: MoE large language models
Abstract (arXiv)
Post-training expert pruning is a promising deployment technique for Mixture-of-Experts (MoE) models. The calibration set is an integral part of the pruning process, however, its correlation with evaluation performance is largely misunderstood and unintuitive. In this work, we present an empirical study of the robustness of calibration perturbations, specifically in MoE models. Showing how calibration size, language, and data distribution affect layer-wise expert pruning for gpt-oss-20b. Our findings reveal that (1) domain-general pruning quality receives minimal benefit from scaling calibration data beyond small sample sizes, (2) cross-lingual calibration transfers effectively, with English-only calibration maintaining strong performance across seven languages of varying typologies and scripts, and (3) a diverse calibration data resembling the pretraining distribution can yield comparable or superior performance to test-distribution data. Ultimately, we provide practical insights as a recipe for better understanding calibration for more effective domain-general expert pruning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 19 of its 23 references that are in the map. First posted on arXiv 11 Oct 2026.
Builds on
19 of 23 references.
Closest ancestors · 1
Same direction, newest first
Other references · 18
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
- 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
6 more
- 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
- 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 79
- 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
- 2016Compression of Neural Machine Translation Models via Pruningcited by 44
- 2025Multilingual Routing in Mixture-of-Expertscited by 29
- 2026Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?cited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
- 2026Toward Calibrated Mixture-of-Experts Under Distribution Shift
- 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
- 2026Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
- 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
- 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
- 2026Higher-order pruning of experts in mixture-of-experts language models
- 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?