mlsanityData to 14 Oct 2026

A Bitter Lesson for Data Filtering

arXiv 2605.19407 ↗First posted 19 May 2026cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

23 of 34 references.

References · 23

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  9. 2023Scaling Data-Constrained Language Modelscited by 403
  10. 2024A Survey on Data Selection for Language Modelscited by 236
  11. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  12. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
11 more
  1. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  2. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  3. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  4. 2017Global Optimality in Low-rank Matrix Optimizationcited by 36
  5. 2025Pre-training under infinite computecited by 29
  6. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  7. 2025When Bad Data Leads to Good Modelscited by 11
  8. 2021An Empirical Exploration in Quality Filtering of Text Datacited by 9
  9. 2022How many labelers do you have? A closer look at gold-standard labelscited by 8
  10. 2025Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Qualitycited by 8
  11. 2025Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretrainingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretrainingcited by 4
  2. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 2
  3. 2026The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Contextcited by 2
  4. 2026Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Datacited by 2
  5. 2026DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Datacited by 1
  6. 2026How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functionscited by 0
  7. 2026How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Textcited by 0
  8. 2026Sequential Pretraining Favors Large Modelscited by 0