# A Bitter Lesson for Data Filtering

arXiv: https://arxiv.org/abs/2605.19407 · HTML: https://mlsanity.com/paper/2605.19407/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 19 May 2026.

## Builds on (most cited first, 23 of 23)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws](https://mlsanity.com/paper/2404.05405/) (2024)
- [Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic](https://mlsanity.com/paper/2404.07177/) (2024)
- [Global Optimality in Low-rank Matrix Optimization](https://mlsanity.com/paper/1702.07945/) (2017)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Do we really have to filter out random noise in pre-training data for language models?](https://mlsanity.com/paper/2502.06604/) (2025)
- [When Bad Data Leads to Good Models](https://mlsanity.com/paper/2505.04741/) (2025)
- [An Empirical Exploration in Quality Filtering of Text Data](https://mlsanity.com/paper/2109.00698/) (2021)
- [How many labelers do you have? A closer look at gold-standard labels](https://mlsanity.com/paper/2206.12041/) (2022)
- [Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality](https://mlsanity.com/paper/2503.07879/) (2025)
- [Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining](https://mlsanity.com/paper/2510.00866/) (2025)

## Built on by (most cited first, 8 of 8)

- [Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining](https://mlsanity.com/paper/2606.16246/) (2026)
- [DataComp-VLM: Improved Open Datasets for Vision-Language Models](https://mlsanity.com/paper/2606.28551/) (2026)
- [The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context](https://mlsanity.com/paper/2607.12963/) (2026)
- [Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data](https://mlsanity.com/paper/2609.11917/) (2026)
- [DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data](https://mlsanity.com/paper/2607.24717/) (2026)
- [How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions](https://mlsanity.com/paper/2605.29448/) (2026)
- [How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text](https://mlsanity.com/paper/2609.40295/) (2026)
- [Sequential Pretraining Favors Large Models](https://mlsanity.com/paper/2610.09611/) (2026)

## Abstract (arXiv)

We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.