mlsanityData to 19 Oct 2026

Pre-training corpora and quality filtering

96 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: language, llms, languages, multilingual, dataset

Key facts. Pre-training corpora and quality filtering: 96 papers on arXiv since 2020 (data through 19 Oct 2026). 34 of its 96 mapped papers (35%) cite “The Pile: An 800GB Dataset of Diverse Text for Language Modeling” (2020, arXiv 2101.00027). 1 new paper 1 Oct–19 Oct.

What’s new

1 Oct – 19 Oct 2026

1 new since 1 Oct

  1. Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

Foundations

Most cited by papers here

  1. The Pile: An 800GB Dataset of Diverse Text for Language Modeling2020cited by 34 here
  2. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale2024cited by 25 here
  3. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research2024cited by 24 here
  4. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only2023cited by 16 here
  5. DataComp-LM: In search of the next generation of training sets for language models2024cited by 12 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 34 here
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 22 here
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 19 here
  4. 2022Training Compute-Optimal Large Language Modelscited by 9 here
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 13 here
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 16 here
  7. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 5 here
  8. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 8 here
  9. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 5 here
15 more it builds on, not in this chain
  1. 2020Scaling Laws for Neural Language Modelscited by 9 here
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 5 here
  3. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 5 here
  4. 2023Data Selection for Language Models via Importance Resamplingcited by 10 here
  5. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 7 here
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 7 here
  7. 2023Scaling Data-Constrained Language Modelscited by 8 here
  8. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 11 here
  9. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 24 here
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 6 here
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 12 here
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 25 here
  13. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 5 here
  14. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 6 here
  15. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 9 here

The 24 most cited of 84.

Browse

30 of 96, ranked 6–35

  1. 6CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages202311
  2. 7Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset20249
  3. 8OLMo: Accelerating the Science of Language Models20246
  4. 9RedPajama: an Open Dataset for Training Large Language Models20246
  5. 10Enhancing Multilingual LLM Pretraining with Model-Based Data Selection20255
  6. 11To Code, or Not To Code? Exploring Impact of Code in Pre-training20245
  7. 12InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training20205
  8. 13Organize the Web: Constructing Domains Enhances Pre-Training Data Curation20254
  9. 14Multilingual Language Model Pretraining using Machine-translated Data20253
  10. 15Code-Switching Curriculum Learning for Multilingual Transfer in LLMs20243
All 30
  1. 16Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources20252
  2. 17Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models20252
  3. 18Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis20242
  4. 19Curriculum Learning for Small Code Language Models20242
  5. 20XLM-E: Cross-lingual Language Model Pre-training via ELECTRA20212
  6. 21FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy20251
  7. 22Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation20251
  8. 23Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training20251
  9. 24Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training20251
  10. 25HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models20251
  11. 26Cleaner Pretraining Corpus Curation with Neural Web Scraping20241
  12. 27FuLG: 150B Romanian Corpus for Language Model Pretraining20241
  13. 28When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale20231
  14. 29CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation20221
  15. 30BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla20211
  16. 31DOBF: A Deobfuscation Pre-Training Objective for Programming Languages20211
  17. 32The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining20260
  18. 33ks-lit-3m: A 3.1 million word kashmiri text dataset for large language model pretraining20260
  19. 34AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages20260
  20. 35PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection20260