mlsanityData to 7 Oct 2026

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

arXiv 2602.14675 ↗First posted 16 Feb 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy. The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual word alignment. We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation. Our analysis reveals that Piedmontese incurs a tokenization penalty relative to higher-resource Romance languages, yet LLMs achieve classification performance approaching that of Italian, French, and English. Machine translation results are asymmetric: models translate adequately from Piedmontese into high-resource languages, but generation into Piedmontese remains challenging. The dataset and code are publicly released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 22 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

11 of 22 references.

References · 11

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  5. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  6. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  7. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  8. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  9. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  10. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  11. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40

Built on by

Not cited yet.