mlsanityData to 7 Oct 2026

"Chi nas dal soch el sent de legn" -- Auditing Text Corpora for Lombard

arXiv 2606.06349 ↗First posted 4 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Several of the world's languages are still under-resourced in terms of Natural Language Processing (NLP) tools. This is mostly due to the lack of high-quality datasets to train, develop, and evaluate systems and models for several tasks, such as Machine Translation (MT). We conduct a manual audit of the parallel and monolingual corpora available for Lombard, an under-resourced language continuum from Italy. Our analysis reveals that the perceived abundance of web-scraped data is an illusion, with massive datasets plagued by severe language misidentification, boilerplate text, and non-linguistic noise. Furthermore, we analyze the orthographic composition of the valid Lombard portions across web-scraped datasets, curated corpora, and benchmarks. Our findings show conflicting orthographical systems and severe representational bias across all corpora: high-quality data is heavily skewed towards Western Lombard varieties, with Eastern ones left on the margins. This underscores the need for variety-aware, community-driven data curation rather than purely quantity-driven scraping.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 27 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

8 of 27 references.

References · 8

  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  2. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  3. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  4. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  5. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  6. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  7. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
  8. 2021XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignmentcited by 15

Built on by

Not cited yet.