"Chi nas dal soch el sent de legn" -- Auditing Text Corpora for Lombard
Not on the map. Why?
Abstract (arXiv)
Several of the world's languages are still under-resourced in terms of Natural Language Processing (NLP) tools. This is mostly due to the lack of high-quality datasets to train, develop, and evaluate systems and models for several tasks, such as Machine Translation (MT). We conduct a manual audit of the parallel and monolingual corpora available for Lombard, an under-resourced language continuum from Italy. Our analysis reveals that the perceived abundance of web-scraped data is an illusion, with massive datasets plagued by severe language misidentification, boilerplate text, and non-linguistic noise. Furthermore, we analyze the orthographic composition of the valid Lombard portions across web-scraped datasets, curated corpora, and benchmarks. Our findings show conflicting orthographical systems and severe representational bias across all corpora: high-quality data is heavily skewed towards Western Lombard varieties, with Eastern ones left on the margins. This underscores the need for variety-aware, community-driven data curation rather than purely quantity-driven scraping.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 27 references that are in the map. First posted on arXiv 4 Jun 2026.
Builds on
8 of 27 references.
References · 8
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
- 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
- 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
- 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
- 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
- 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
- 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
- 2021XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignmentcited by 15
Built on by
Not cited yet.