# Pre-training corpora and quality filtering

Area: [LLM pre-training data and scaling laws](https://mlsanity.com/pretrain/index.md). HTML: https://mlsanity.com/pretrain/pre-training-corpora-and-quality-filtering/

## Key facts

- Pre-training corpora and quality filtering: 96 papers on arXiv since 2020 (data through 19 Oct 2026).
- 34 of its 96 mapped papers (35%) cite “The Pile: An 800GB Dataset of Diverse Text for Language Modeling” (2020, arXiv 2101.00027).
- 25 of its 96 mapped papers (26%) cite “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale” (2024, arXiv 2406.17557).
- 1 new papers 1 Oct–19 Oct.

## Most cited here

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020), cited by 34 papers here
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024), cited by 25 papers here
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024), cited by 24 papers here
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023), cited by 16 papers here
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024), cited by 12 papers here
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023), cited by 11 papers here
- [Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://mlsanity.com/paper/2412.02595/) (2024), cited by 9 papers here
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024), cited by 6 papers here
- [RedPajama: an Open Dataset for Training Large Language Models](https://mlsanity.com/paper/2411.12372/) (2024), cited by 6 papers here
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020), cited by 5 papers here

## New (1 Oct–19 Oct)

- [Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection](https://mlsanity.com/paper/2610.11585/) (2026-10-08)