mlsanityData to 7 Oct 2026

Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

arXiv 2201.06642 ↗First posted 17 Jan 2022cs.CLcited by 143

Not on the map. Why?

Abstract (arXiv)

The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 17 Jan 2022.

Builds on

28 of 48 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
16 more
  1. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2019Green AIcited by 620
  4. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  5. 2019CamemBERT: a Tasty French Language Modelcited by 368
  6. 2016FastText.zip: Compressing text classification modelscited by 331
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  8. 2018Learning Word Vectors for 157 Languagescited by 279
  9. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  10. 2018A Simple Method for Commonsense Reasoningcited by 223
  11. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  12. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  13. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  14. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  15. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  16. 2013Polyglot: Distributed Word Representations for Multilingual NLPcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 29; 2024: 55; 2025: 30; 2026: 18.

143 papers.

Sort:

By area:

  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  5. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  6. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  7. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  8. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  9. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  10. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  11. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
  12. 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
38 more
  1. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  2. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34
  3. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  4. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  5. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  6. 2023Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*cited by 24
  7. 2023LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Languagecited by 24
  8. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulationcited by 24
  9. 2023ChatGPT MT: Competitive for High- (but not Low-) Resource Languagescited by 23
  10. 2022Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Datacited by 22
  11. 2023CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Datacited by 21
  12. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  13. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 19
  14. 2024HyperCLOVA X Technical Reportcited by 18
  15. 2023A Survey of Corpora for Germanic Low-Resource Languages and Dialectscited by 16
  16. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  17. 2024Gl'orIA -- A Generative and Open Large Language Model for Portuguesecited by 16
  18. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  19. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  20. 2023Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Modelcited by 14
  21. 2024TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generationcited by 13
  22. 2024Multilingual Large Language Models and Curse of Multilingualitycited by 13
  23. 2024Do Multilingual Large Language Models Mitigate Stereotype Bias?cited by 13
  24. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  25. 2025Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translationcited by 13
  26. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 13
  27. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  28. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  29. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  30. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  31. 2025Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basquecited by 12
  32. 2024Building a Large Japanese Web Corpus for Large Language Modelscited by 11
  33. 2024Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysiscited by 11
  34. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 11
  35. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  36. 2024Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarkingcited by 10
  37. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  38. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9

50 most cited of 143 All 143 (JSON)