mlsanityData to 14 Oct 2026

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models

arXiv 2410.05661 ↗First posted 8 Oct 2024cs.LG cs.AIcited by 25

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferability and discrepancies of scaling laws between Dense Models and Mixture of Experts (MoE) models. Through a combination of theoretical analysis and extensive experiments, including consistent loss scaling, optimal batch size and learning rate scaling, and resource allocation strategies scaling, our findings reveal that the power-law scaling framework also applies to MoE Models, indicating that the fundamental principles governing the scaling behavior of these models are preserved, even though the architecture differs. Additionally, MoE Models demonstrate superior generalization, resulting in lower testing losses with the same training compute budget compared to Dense Models. These findings indicate the scaling consistency and transfer generalization capabilities of MoE Models, providing new insights for optimizing MoE Model training and deployment strategies.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 42 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

25 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  11. 2021Explaining Neural Scaling Lawscited by 363
  12. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
10 more
  1. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  2. 2018An Empirical Model of Large-Batch Trainingcited by 281
  3. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  4. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  5. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  6. 2021Pareto Self-Supervised Training for Few-Shot Learningcited by 52
  7. 2022Using natural language prompts for machine translationcited by 27
  8. 2023Learning How to Propagate Messages in Graph Neural Networkscited by 23
  9. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  10. 2021Hyperparameter Transfer Learning with Adaptive Complexitycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 57
  2. 2025Scaling Laws for Optimal Data Mixturescited by 52
  3. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  4. 2025Scaling Laws for Native Multimodal Modelscited by 50
  5. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  6. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
  7. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  8. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  9. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compressioncited by 7
  10. 2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
  11. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  12. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
13 more
  1. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  2. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  3. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimizationcited by 2
  4. 2026Slicing and Dicing: Configuring Optimal Mixtures of Expertscited by 2
  5. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  6. 2026Revisiting the Shape Convention of Transformer Language Modelscited by 1
  7. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  8. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1
  9. 2025Utilizing dynamic sparsity on pretrained DETRcited by 0
  10. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Expertscited by 0
  11. 2026Hyperparameter Scaling Laws Across MoE Sparsitycited by 0
  12. 2026X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depthscited by 0
  13. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Laws for Fine-Grained Mixture of Experts
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  3. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  4. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  5. 2025Mixture of Experts in Large Language Models
  6. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2025Advancing Expert Specialization for Better MoE