mlsanityData to 14 Oct 2026

The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts

arXiv 2505.06839 ↗First posted 11 May 2025cs.LG cs.AI stat.MLcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 11 of its 27 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

11 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models21 Jan
  2. 2024Mixture of Parrots: Experts improve memorization more than reasoning24 Oct
  3. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model23 May
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 4

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 253

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025The Mean-Field Dynamics of Transformerscited by 30
  2. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  3. 2026STEM: Scaling Transformers with Embedding Modulescited by 12
  4. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
  5. 2025Secret mixtures of experts inside your LLMcited by 2
  6. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  7. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0
  8. 2026FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approachcited by 0
  9. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  2. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  3. 2026Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
  4. 2026Towards a Statistical Understanding of Mixture-of-Experts
  5. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  6. 2025Sparsity and Superposition in Mixture of Experts
  7. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  8. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute