# Scaling laws and compute-optimal training

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/scaling-laws-and-compute-optimal-training/

## Key facts

- Scaling laws and compute-optimal training: 87 papers on arXiv since 2020 (data through 19 Oct 2026).
- 78 of its 87 mapped papers (90%) cite “Scaling Laws for Neural Language Models” (2020, arXiv 2001.08361).
- 31 of its 87 mapped papers (36%) cite “Scaling Laws for Autoregressive Generative Modeling” (2020, arXiv 2010.14701).
- 2 new papers 1 Oct–19 Oct.

## Most cited here

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020), cited by 78 papers here
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020), cited by 31 papers here
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023), cited by 26 papers here
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021), cited by 19 papers here
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024), cited by 18 papers here
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023), cited by 16 papers here
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024), cited by 14 papers here
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022), cited by 12 papers here
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023), cited by 12 papers here
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024), cited by 12 papers here

## New (1 Oct–19 Oct)

- [Compute-Optimal Scaling Laws for Spiking Language Models](https://mlsanity.com/paper/2610.15146/) (2026-10-10)
- [Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models](https://mlsanity.com/paper/2610.06387/) (2026-10-05)