mlsanityData to 7 Oct 2026

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

arXiv 2501.12370 ↗First posted 21 Jan 2025cs.LG cs.AIcited by 48

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the compute per example. While scaling typically involves increasing both, the precise interplay between these factors and their combined contribution to overall capacity remains not fully understood. We explore this relationship in the context of sparse Mixture-of-Experts (MoEs), which allow scaling the number of parameters without proportionally increasing the FLOPs per example. We investigate how varying the sparsity level, i.e., the fraction of inactive parameters, impacts model's performance during pretraining and downstream few-shot evaluation. We find that under different constraints (e.g., parameter size and total training compute), there is an optimal level of sparsity that improves both training efficiency and model performance. These results provide a better understanding of the impact of sparsity in scaling laws for MoEs and complement existing works in this area, offering insights for designing more efficient architectures.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 43 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

36 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Mixture of Parrots: Experts improve memorization more than reasoning24 Oct
  2. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models8 Oct
  3. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models3 Apr
  4. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
16 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2018CoQA: A Conversational Question Answering Challengecited by 674
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2018Universal Transformerscited by 613
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  6. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  7. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  8. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  10. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  11. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  12. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  13. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  14. 2024Mixture of A Million Expertscited by 70
  15. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  16. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 26.

48 papers.

Sort:

By area:

  1. 2025Distillation Scaling Lawscited by 59
  2. 2025Scaling Laws for Optimal Data Mixturescited by 49
  3. 2025Scaling Laws for Native Multimodal Modelscited by 47
  4. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  5. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  6. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  7. 2025Scaling Inference-Efficient Language Modelscited by 14
  8. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 13
  9. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  10. 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
  11. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 9
  12. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 7
36 more
  1. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  2. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compressioncited by 7
  3. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inferencecited by 7
  4. 2025Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Expertscited by 5
  5. 2025Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Modelscited by 5
  6. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  7. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  8. 2025Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMscited by 4
  9. 2026When Does Sparsity Mitigate the Curse of Depth in LLMscited by 4
  10. 2026Loop the Loopies!cited by 4
  11. 2025Soup-of-Experts: Pretraining Specialist Models via Parameters Averagingcited by 3
  12. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  13. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 3
  14. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Expertscited by 3
  15. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  16. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelismcited by 2
  17. 2026On Neural Scaling Laws for Weather Emulation through Continual Trainingcited by 2
  18. 2026Slicing and Dicing: Configuring Optimal Mixtures of Expertscited by 2
  19. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  20. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  21. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimizationcited by 1
  22. 2026LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Modelscited by 1
  23. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Modelscited by 0
  24. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Taskscited by 0
  25. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraintscited by 0
  26. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Designcited by 0
  27. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 0
  28. 2026Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matchingcited by 0
  29. 2026Breakeven complexity: A new perspective on neural partial differential equation solverscited by 0
  30. 2026Asymmetric Scaling Laws from Sparse Featurescited by 0
  31. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Expertscited by 0
  32. 2026Dynamic HBM Repartitioning for Multi-Turn MoE Servingcited by 0
  33. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  34. 2026X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depthscited by 0
  35. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
  36. 2026Scaling Laws for Looped Mixture of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  2. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
  3. 2026Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
  4. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  6. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Models
  7. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  8. 2025Sparsity and Superposition in Mixture of Experts