mlsanityData to 7 Oct 2026

Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

arXiv 2609.07699 ↗First posted 7 Sep 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

18 of 29 references.

References · 18

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  7. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  8. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  9. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  10. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  11. 2025Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corporacited by 7
  12. 2025FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filteringcited by 6
6 more
  1. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
  2. 2025Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Modelscited by 6
  3. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  4. 2026ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMscited by 3
  5. 2024SWEb: A Large Web Dataset for the Scandinavian Languagescited by 1
  6. 2026P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMscited by 1

Built on by

Not cited yet.