mlsanityData to 14 Oct 2026

Generalizing Scaling Laws for Dense and Sparse Large Language Models

arXiv 2508.06617 ↗First posted 8 Aug 2025cs.LG cs.AI cs.PFcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Despite recent advancements of large language models (LLMs), optimally predicting the model size for LLM pretraining or allocating optimal resources still remains a challenge. Several efforts have addressed the challenge by proposing different empirical scaling laws, but almost all of them are architecture-specific (dense or sparse). In this work we revisit existing empirical scaling laws and propose a generalized scaling law to provide a unified framework that is applicable to both dense and sparse large language models. We evaluate and compare our proposed scaling law with existing scaling laws and demonstrate that our proposed scaling law captures the scaling behavior of existing scaling laws. Further, we show an IsoFLOP comparison between our proposed scaling law and the state-of-the-art scaling law to illustrate the effectiveness of our proposed scaling law for Mixture-of-Expert (MoE)-based very large LLMs like DeepSeek-V3. Our proposed scaling law can be used to estimate the best model hyperparameters (Model size, Tokens and Compute) for a given sparsity or to identify the optimal sparsity for the given model hyperparameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 37 of its 43 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

37 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models21 Jan
  2. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  4. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  6. 2024Mixtral of Experts8 Jan
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
17 more
  1. 20242 OLMo 2 Furiouscited by 340
  2. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  4. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  5. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  6. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  7. 2024Scaling Laws for Precisioncited by 95
  8. 2025Distillation Scaling Lawscited by 67
  9. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43
  10. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
  11. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
  12. 2021Scaling Laws for Acoustic Modelscited by 21
  13. 2025Scaling Law for Quantization-Aware Trainingcited by 17
  14. 2023ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scalescited by 13
  15. 2024Efficient Autoregressive Audio Modeling via Next-Scale Predictioncited by 11
  16. 2021libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculationscited by 3
  17. 2023Autotuning Apache TVM-based Scientific Applications Using Bayesian Optimizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  2. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  3. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  4. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
  5. 2025Pruning Laws for Large Language Models
  6. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
  7. 2025Compression Scaling Laws:Unifying Sparsity and Quantization
  8. 2026Scaling Embeddings Outperforms Scaling Experts in Language Models