mlsanityData to 14 Oct 2026

Unified Scaling Laws for Routed Language Models

arXiv 2202.01169 ↗First posted 2 Feb 2022cs.CL cs.LGcited by 253

Not on the map. Why?

Cited by 130 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter count and computational requirement form two independent axes along which an increase leads to better performance. In this work we derive and justify scaling laws defined on these two variables which generalize those known for standard language models and describe the performance of a wide range of routing architectures trained via three different techniques. Afterwards we provide two applications of these laws: first deriving an Effective Parameter Count along which all models scale at the same rate, and then using the scaling coefficients to give a quantitative comparison of the three routing techniques considered. Our analysis derives from an extensive evaluation of Routing Networks across five orders of magnitude of size, including models with hundreds of experts and hundreds of billions of parameters.

Key facts. Cited by 253 papers in the map (data through 14 Oct 2026). Builds on 39 of its 69 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

39 of 69 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
27 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  7. 2018Computational Optimal Transportcited by 780
  8. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  10. 2021Explaining Neural Scaling Lawscited by 363
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  12. 2018An Empirical Model of Large-Batch Trainingcited by 281
  13. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  14. 2021Scaling Laws for Transfercited by 272
  15. 2021Hash Layers For Large Sparse Modelscited by 237
  16. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  17. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  18. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  19. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  20. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  21. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  22. 2021Scaling Laws for Neural Machine Translationcited by 111
  23. 2021Learning Curve Theorycited by 85
  24. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  25. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 50
  26. 2021On Anytime Learning at Macroscalecited by 20
  27. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 22; 2023: 37; 2024: 70; 2025: 57; 2026: 67.

253 papers.

Sort:

By area:

  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2024Mixtral of Expertscited by 1,823
  4. 2024Scaling and evaluating sparse autoencoderscited by 553
  5. 2023The Falcon Series of Open Language Modelscited by 535
  6. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  7. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  9. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  11. 2023From Sparse to Soft Mixtures of Expertscited by 248
  12. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
38 more
  1. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  4. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  6. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  7. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  8. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  9. 2022Broken Neural Scaling Lawscited by 109
  10. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  11. 2024Scaling Laws for Precisioncited by 95
  12. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  13. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  14. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  15. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  16. 2023Modular Deep Learningcited by 83
  17. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  18. 2024Mixture of A Million Expertscited by 72
  19. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  20. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  21. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 64
  22. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  23. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  24. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  25. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  26. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  27. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 59
  28. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  29. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  30. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  31. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  32. 2025Scaling Laws for Optimal Data Mixturescited by 52
  33. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  34. 2025Scaling Laws for Native Multimodal Modelscited by 50
  35. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  36. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  37. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  38. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43

50 most cited of 253 All 253 (JSON)