mlsanityData to 14 Oct 2026

Will we run out of data? Limits of LLM scaling based on human-generated data

arXiv 2211.04325 ↗First posted 26 Oct 2022cs.LG cs.AI cs.CLcited by 195

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.

Key facts. Cited by 195 papers in the map (data through 14 Oct 2026). Builds on 25 of its 105 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

25 of 105 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  9. 2019Solving Rubik's Cube with a Robot Handcited by 734
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  11. 2022Large Language Models Can Self-Improvecited by 664
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
13 more
  1. 2018AI safety via debatecited by 400
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  3. 2021Scaling Laws for Transfercited by 272
  4. 2021Mastering Atari Games with Limited Datacited by 237
  5. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  6. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  7. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  8. 2022Language Models Can Teach Themselves to Program Bettercited by 78
  9. 2021Curriculum learning for language modelingcited by 32
  10. 2021An Empirical Exploration in Quality Filtering of Text Datacited by 9
  11. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  12. 2017Social Emotion Mining Techniques for Facebook Posts Reaction Predictioncited by 3
  13. 2015Investigation of the widely applicable Bayesian information criterioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 23; 2024: 32; 2025: 90; 2026: 50.

195 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  3. 2023The Falcon Series of Open Language Modelscited by 535
  4. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  5. 2023Scaling Data-Constrained Language Modelscited by 403
  6. 2023Challenges and Applications of Large Language Modelscited by 254
  7. 2023Pretraining Language Models with Human Preferencescited by 247
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  9. 2025Hierarchical Reasoning Modelcited by 177
  10. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  11. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  12. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
38 more
  1. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  2. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  3. 2025Self-Adapting Language Modelscited by 84
  4. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  5. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  6. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  7. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 63
  8. 2024Synthetic continued pretrainingcited by 60
  9. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  10. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  11. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 53
  12. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  13. 2025Reasoning to Learn from Latent Thoughtscited by 45
  14. 2025Language Self-Play For Data-Free Trainingcited by 43
  15. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  16. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  17. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  18. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  19. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  20. 2025Parallel Scaling Law for Language Modelscited by 30
  21. 2025Pre-training under infinite computecited by 29
  22. 2025Leaky Thoughts: Large Reasoning Models Are Not Private Thinkerscited by 28
  23. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  24. 2024TransformerFAM: Feedback attention is working memorycited by 22
  25. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 21
  26. 2024A social path to human-like artificial intelligencecited by 20
  27. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  28. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  29. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 17
  30. 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 16
  31. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  32. 2023A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformerscited by 15
  33. 2023Market Concentration Implications of Foundation Modelscited by 15
  34. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectivecited by 15
  35. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  36. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 15
  37. 2026Training Language Models via Neural Cellular Automatacited by 15
  38. 2023EarthPT: a time series foundation model for Earth Observationcited by 14

50 most cited of 195 All 195 (JSON)