mlsanityData to 19 Oct 2026

Scaling laws and compute-optimal training

87 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: scaling, laws, scaling laws, law, size

Key facts. Scaling laws and compute-optimal training: 87 papers on arXiv since 2020 (data through 19 Oct 2026). 78 of its 87 mapped papers (90%) cite “Scaling Laws for Neural Language Models” (2020, arXiv 2001.08361). 2 new papers 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

2 new since 1 Oct

  1. Compute-Optimal Scaling Laws for Spiking Language Models also fits: Optimizers and stability in LLM pre-training
  2. Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models

Foundations

Most cited by papers here

  1. Scaling Laws for Neural Language Models2020cited by 78 here
  2. Scaling Laws for Autoregressive Generative Modeling2020cited by 31 here
  3. Scaling Data-Constrained Language Models2023cited by 26 here
  4. Scaling Language Models: Methods, Analysis & Insights from Training Gopher2021cited by 19 here
  5. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies2024cited by 18 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 20 here
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 7 here
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 19 here
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 10 here
  5. 2022Training Compute-Optimal Large Language Modelscited by 47 here
  6. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 16 here
  7. 2023Scaling Data-Constrained Language Modelscited by 26 here
  8. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 10 here
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 18 here
  10. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 9 here
  11. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 6 here
  12. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 14 here
12 more it builds on, not in this chain
  1. 2020Scaling Laws for Neural Language Modelscited by 78 here
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 31 here
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 9 here
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 12 here
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 12 here
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 8 here
  7. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 7 here
  8. 2024A Dynamical Model of Neural Scaling Lawscited by 12 here
  9. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 7 here
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 7 here
  11. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 9 here
  12. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 8 here

The 24 most cited of 92.

Browse

30 of 87, ranked 6–35

  1. 6Scaling Laws for Generative Mixed-Modal Language Models202316
  2. 7Resolving Discrepancies in Compute-Optimal Scaling of Language Models202414
  3. 8A Dynamical Model of Neural Scaling Laws202412
  4. 9Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling202312
  5. 10Beyond neural scaling laws: beating power law scaling via data pruning202212
  6. 11Unified Scaling Laws for Routed Language Models202210
  7. 12Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations20249
  8. 13Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies20249
  9. 14Reconciling Kaplan and Chinchilla Scaling Laws20246
  10. 15Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster20235
All 30
  1. 16Scaling Laws for Sparsely-Connected Foundation Models20235
  2. 17Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining20254
  3. 18Improved Scaling Laws in Linear Regression via Data Reuse20254
  4. 19Scaling Law for Language Models Training Considering Batch Size20244
  5. 20Data Scaling Laws in NMT: The Effect of Noise and Architecture20224
  6. 21(Mis)Fitting: A Survey of Scaling Laws20253
  7. 22Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression20253
  8. 23Scaling Laws for Multilingual Language Models20243
  9. 24Towards Robust Scaling Laws for Optimizers20262
  10. 25Prescriptive Scaling Laws for Data Constrained Training20262
  11. 26Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models20252
  12. 27Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay20261
  13. 28Scaling Laws for Differentially Private Language Models20251
  14. 29LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws20251
  15. 30Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training20251
  16. 31Pretraining Scaling Laws for Generative Evaluations of Language Models20251
  17. 32Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining20251
  18. 33Relative-Based Scaling Law for Neural Language Models20251
  19. 34gzip Predicts Data-dependent Scaling Laws20241
  20. 35Evidence of a log scaling law for political persuasion with large language models20241