mlsanityData to 14 Oct 2026

GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages

arXiv 2410.23825 ↗First posted 31 Oct 2024cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languages with large dominant communities. However, there is no corpus available that (i) covers a wide range of minority languages; (ii) is generated by an open-source reproducible pipeline; and (iii) is rigorously cleaned from noise, making it trustworthy to use. We present GlotCC, a clean, document-level, 2TB general domain corpus derived from CommonCrawl, covering more than 1000 languages. We make GlotCC and the system used to generate it - including the pipeline, language identification model, and filters - available to the research community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1, Pipeline v. 3.0 https://github.com/cisnlp/GlotCC.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 44 of its 94 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

44 of 94 references.

References · 44

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2018Model Cards for Model Reportingcited by 1,080
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
32 more
  1. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  2. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  3. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 353
  4. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  5. 2018Learning Word Vectors for 157 Languagescited by 280
  6. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  7. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  8. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  9. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  10. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  11. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  12. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  13. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  14. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  15. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  16. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  17. 2023An Open Dataset and Model for Language Identificationcited by 50
  18. 2020Mapping Languages: The Corpus of Global Language Usecited by 42
  19. 2018Automatic Language Identification in Texts: A Surveycited by 41
  20. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  21. 2022AfroLID: A Neural Language Identification Tool for African Languagescited by 39
  22. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  23. 2023MasakhaNEWS: News Topic Classification for African languagescited by 31
  24. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  25. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  26. 2021AI4D -- African Language Programcited by 18
  27. 2023GlotScript: A Resource and Tool for Low Resource Writing System Identificationcited by 15
  28. 2023Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corporacited by 12
  29. 2024MaskLID: Code-Switching Language Identification through Iterative Maskingcited by 11
  30. 2017Natural Language Processing with Small Feed-Forward Networkscited by 10
  31. 2024Geographically-Informed Language Identificationcited by 6
  32. 2017Open-Set Language Identificationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  2. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  3. 2025ConLID: Supervised Contrastive Learning for Low-Resource Language Identificationcited by 6
  4. 2025Ibom NLP: A Step Toward Inclusive Natural Language Processing for Nigeria's Minority Languagescited by 6
  5. 2025DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detectioncited by 5
  6. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languagescited by 3
  7. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scriptscited by 2
  8. 2025Improving Informally Romanized Language Identificationcited by 1
  9. 2025Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Modelscited by 1
  10. 2026Assessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Modelscited by 1
  11. 2025On Multilingual Encoder Language Model Compression for Low-Resource Languagescited by 0
  12. 2026PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Developmentcited by 0
4 more
  1. 2026The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoningcited by 0
  2. 2026Improving Language Identification for Code-Switched Utterances with Integer Linear Programmingcited by 0
  3. 2026Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Modelscited by 0
  4. 2026BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Itemscited by 0