mlsanityData to 7 Oct 2026

Does Corpus Quality Really Matter for Low-Resource Languages?

arXiv 2203.08111 ↗First posted 15 Mar 2022cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

The vast majority of non-English corpora are derived from automatically filtered versions of CommonCrawl. While prior work has identified major issues on the quality of these datasets (Kreutzer et al., 2021), it is not clear how this impacts downstream performance. Taking representation learning in Basque as a case study, we explore tailored crawling (manually identifying and scraping websites with high-quality content) as an alternative to filtering CommonCrawl. Our new corpus, called EusCrawl, is similar in size to the Basque portion of popular multilingual corpora like CC100 and mC4, yet it has a much higher quality according to native annotators. For instance, 66% of documents are rated as high-quality for EusCrawl, in contrast with <33% for both mC4 and CC100. Nevertheless, we obtain similar results on downstream NLU tasks regardless of the corpus used for pre-training. Our work suggests that NLU performance in low-resource languages is not primarily constrained by the quality of the data, and other factors like corpus size and domain coverage can play a more important role.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 12 of its 20 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

12 of 20 references.

References · 12

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2019How multilingual is Multilingual BERT?cited by 642
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  11. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  12. 2020Give your Text Representation Models some Love: the Case for Basquecited by 28

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 5; 2024: 1; 2025: 2.

11 papers.

  1. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  2. 2022PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generationcited by 22
  3. 2023A Survey of Corpora for Germanic Low-Resource Languages and Dialectscited by 16
  4. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  5. 2022Lessons learned from the evaluation of Spanish Language Modelscited by 8
  6. 2024The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspectivecited by 8
  7. 2023How User Language Affects Conflict Fatality Estimates in ChatGPTcited by 4
  8. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  9. 2022Uzbek Sentiment Analysis based on local Restaurant Reviewscited by 3
  10. 2023Bridging Natural Language Processing and Psycholinguistics: computationally grounded semantic similarity datasets for Basque and Spanishcited by 0
  11. 2025Euskarazko lehen C1 ebaluatzaile automatikoacited by 0