mlsanityData to 7 Oct 2026

CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

arXiv 1911.00359 ↗First posted 1 Nov 2019cs.CL cs.IR cs.LGcited by 544

Not on the map. Why?

Abstract (arXiv)

Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia.

Key facts. Cited by 544 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 60; 2022: 73; 2023: 94; 2024: 157; 2025: 97; 2026: 63.

544 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  11. 2023A Comprehensive Overview of Large Language Modelscited by 639
  12. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
38 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  4. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  5. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  6. 2023Scaling Data-Constrained Language Modelscited by 391
  7. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  8. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  9. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  10. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  11. 2023Data Filtering Networkscited by 247
  12. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  13. 2021Internet-Augmented Dialogue Generationcited by 217
  14. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  15. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  16. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  17. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  18. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  19. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  20. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  21. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  22. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  23. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  24. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  25. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  26. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  27. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  28. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  29. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  30. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  31. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  32. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 123
  33. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  34. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  35. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  36. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  37. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  38. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103

50 most cited of 544 All 544 (JSON)