# When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale

arXiv: https://arxiv.org/abs/2309.04564 · HTML: https://mlsanity.com/paper/2309.04564/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 48 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 2 of 2)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)

## Built on by (most cited first, 10 of 10)

- [QuRating: Selecting High-Quality Data for Training Language Models](https://mlsanity.com/paper/2402.09739/) (2024)
- [Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning](https://mlsanity.com/paper/2502.07154/) (2025)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [A Survey of LLM × DATA](https://mlsanity.com/paper/2505.18458/) (2025)
- [Unified Data Selection for LLM Reasoning](https://mlsanity.com/paper/2605.22389/) (2026)
- [Quanda: An Interpretability Toolkit for Training Data Attribution Evaluation and Beyond](https://mlsanity.com/paper/2410.07158/) (2024)
- [Uncertainties of Latent Representations in Computer Vision](https://mlsanity.com/paper/2408.14281/) (2024)
- [Reasoning Quality Emerges Early: Data Curation for Reasoning Models](https://mlsanity.com/paper/2606.26797/) (2026)
- [MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation](https://mlsanity.com/paper/2608.29623/) (2026)
- [SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought](https://mlsanity.com/paper/2609.33181/) (2026)

## Abstract (arXiv)

Large volumes of text data have contributed significantly to the development of large language models (LLMs) in recent years. This data is typically acquired by scraping the internet, leading to pretraining datasets comprised of noisy web text. To date, efforts to prune these datasets down to a higher quality subset have relied on hand-crafted heuristics encoded as rule-based filters. In this work, we take a wider view and explore scalable estimates of data quality that can be used to systematically measure the quality of pretraining data. We perform a rigorous comparison at scale of the simple data quality estimator of perplexity, as well as more sophisticated and computationally intensive estimates of the Error L2-Norm and memorization. These metrics are used to rank and prune pretraining corpora, and we subsequently compare LLMs trained on these pruned datasets. Surprisingly, we find that the simple technique of perplexity outperforms our more computationally expensive scoring methods. We improve over our no-pruning baseline while training on as little as 30% of the original training dataset. Our work sets the foundation for unexplored strategies in automatically curating high quality corpora and suggests the majority of pretraining data can be removed while retaining performance.