mlsanityData to 7 Oct 2026

Language models scale reliably with over-training and on downstream tasks

arXiv 2403.08540 ↗First posted 13 Mar 2024cs.CL cs.LGcited by 105

Not on the map. Why?

Abstract (arXiv)

Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32× over-trained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)unicodex2014each from experiments that take 300× less compute. Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20× less compute. Our experiments are available at https://github.com/mlfoundations/scaling.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 105 of its 128 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

105 of 128 references.

References · 105

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  13. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  14. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  15. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  16. 2022Emergent Abilities of Large Language Modelscited by 2,483
  17. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  18. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  19. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  20. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  21. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  22. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  23. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  24. 2020GLU Variants Improve Transformercited by 1,724
  25. 2019Root Mean Square Layer Normalizationcited by 1,425
  26. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  27. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  28. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  29. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  30. 2021Scaling Vision Transformerscited by 993
  31. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  32. 2021Ethical and social risks of harm from Language Modelscited by 957
  33. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  34. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  35. 2023StarCoder: may the source be with you!cited by 790
  36. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  37. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  38. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 41; 2026: 36.

105 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  3. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  4. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  5. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  6. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  7. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  8. 2024Scaling Laws for Precisioncited by 92
  9. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  10. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  11. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  12. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
38 more
  1. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  2. 2025Distillation Scaling Lawscited by 59
  3. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  4. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  5. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  6. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  7. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  8. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  9. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  10. 2024Establishing Task Scaling Laws via Compute-Efficient Model Ladderscited by 32
  11. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  12. 2024Not All LLM Reasoners Are Created Equalcited by 29
  13. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 27
  14. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  15. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Modelscited by 24
  16. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  17. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 23
  18. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 23
  19. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  20. 2024AutoScale: Scale-Aware Data Mixing for Pre-Training LLMscited by 20
  21. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  22. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  23. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  24. 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 15
  25. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  26. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  27. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 14
  28. 2025Scaling Inference-Efficient Language Modelscited by 14
  29. 2025Value-Based Deep RL Scales Predictablycited by 14
  30. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  31. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  32. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  33. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 12
  34. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  35. 2026Prescriptive Scaling Laws for Data Constrained Trainingcited by 9
  36. 2024ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Languagecited by 8
  37. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 8
  38. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 8

50 most cited of 105 All 105 (JSON)