mlsanityData to 7 Oct 2026

Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling

arXiv 2501.13779 ↗First posted 23 Jan 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

While Large Language Models require more and more data to train and scale, rather than looking for any data to acquire, we should consider what types of tasks are more likely to benefit from data scaling. We should be intentional in our data acquisition. We argue that the shape of the data itself, such as its compositional and structural patterns, informs which tasks to prioritize in data scaling, and shapes the development of the next generation of compute paradigms for tasks where data scaling is inefficient, or even insufficient.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 23 Jan 2025.

Builds on

16 of 28 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  7. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  9. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  10. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  11. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  12. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
4 more
  1. 2024Distilling System 2 into System 1cited by 152
  2. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  3. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  4. 2024Topological Data Analysis Applications in Natural Language Processing: A Surveycited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  2. 2025Influence Functions for Efficient Data Selection in Reasoningcited by 10