mlsanityData to 14 Oct 2026

Evaluating Expert Contributions in a MoE LLM for Quiz-Based Tasks

arXiv 2502.17187 ↗First posted 24 Feb 2025cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Recently, Large Language Models (LLMs) with Mixture of Experts (MoE) layers have gained significant attention. Currently, state-of-the-art LLMs utilize this architecture. There is a substantial amount of research on how to train such models and how to select hyperparameters for this architecture. However, there is a lack of studies focusing on post-evaluation analysis of MoE layer properties. In this paper, we take a first step toward closing this gap by evaluating expert contributions on the quiz-based MMLU benchmark. We show that most experts were never activated during inference on this benchmark. Additionally, the output distribution of gating networks is much closer to uniform than sparse. Finally, we demonstrate that the average performance of some experts within the same layer varies significantly.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 6 of its 8 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

6 of 8 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DeepSeek-V3 Technical Report27 Dec
  2. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  3. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 2

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Efficient Large Language Models: A Surveycited by 186

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Mixture of Experts in Large Language Models
  2. 2025Mixture of Experts in Large Language Models
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  6. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  7. 2025Advancing Expert Specialization for Better MoE
  8. 2026REAM: Merging Improves Pruning of Experts in LLMs