mlsanityData to 7 Oct 2026

Scaling Data-Constrained Language Models

arXiv 2305.16264 ↗First posted 25 May 2023cs.CL cs.AI cs.LGcited by 391

Not on the map. Why?

Abstract (arXiv)

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

Key facts. Cited by 391 papers in the map (data through 7 Oct 2026). Builds on 103 of its 139 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

103 of 139 references.

References · 103

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  13. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  15. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  16. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  17. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  18. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  19. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  20. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  21. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  22. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  24. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  25. 2023LIMA: Less Is More for Alignmentcited by 1,080
  26. 2021Ethical and social risks of harm from Language Modelscited by 957
  27. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  28. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  29. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  30. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  31. 2023StarCoder: may the source be with you!cited by 790
  32. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  33. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  34. 2022Galactica: A Large Language Model for Sciencecited by 736
  35. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  36. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  37. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  38. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 131; 2025: 125; 2026: 113.

391 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2023Textbooks Are All You Needcited by 561
  6. 2023The Falcon Series of Open Language Modelscited by 526
  7. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  9. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  10. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  11. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  12. 2024Generative Representational Instruction Tuningcited by 227
38 more
  1. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  2. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  3. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  4. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  5. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  6. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  7. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  8. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  9. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  10. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  11. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  12. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  13. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  14. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  15. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  16. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  17. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 93
  18. 2024Scaling Laws for Precisioncited by 92
  19. 2024How to Train Data-Efficient LLMscited by 91
  20. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  21. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  22. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  23. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  24. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  25. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  26. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  27. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  28. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  29. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  30. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  31. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  32. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 59
  33. 2025Distillation Scaling Lawscited by 59
  34. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  35. 2024Synthetic continued pretrainingcited by 56
  36. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  37. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  38. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54

50 most cited of 391 All 391 (JSON)