mlsanityData to 14 Oct 2026

esCorpius: A Massive Spanish Crawling Corpus

arXiv 2206.15147 ↗First posted 30 Jun 2022cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

In the recent years, transformer-based models have lead to significant advances in language modelling for natural language processing. However, they require a vast amount of data to be (pre-)trained and there is a lack of corpora in languages other than English. Recently, several initiatives have presented multilingual datasets obtained from automatic web crawling. However, the results in Spanish present important shortcomings, as they are either too small in comparison with other languages, or present a low quality derived from sub-optimal cleaning and deduplication. In this paper, we introduce esCorpius, a Spanish crawling corpus obtained from near 1 Pb of Common Crawl data. It is the most extensive corpus in Spanish with this level of quality in the extraction, purification and deduplication of web textual content. Our data curation process involves a novel highly parallel cleaning pipeline and encompasses a series of deduplication mechanisms that together ensure the integrity of both document and paragraph boundaries. Additionally, we maintain both the source web page URL and the WARC shard origin URL in order to complain with EU regulations. esCorpius has been released under CC BY-NC-ND 4.0 license and is available on HuggingFace.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

15 of 23 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  9. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  10. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  11. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  12. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
3 more
  1. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  2. 2021MarIA: Spanish Language Modelscited by 36
  3. 2022IT5: Text-to-text Pretraining for Italian Language Understanding and Generationcited by 20

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 31