# Modeling Orthographic Variation in Occitan's Dialects

arXiv: https://arxiv.org/abs/2404.19315 · HTML: https://mlsanity.com/paper/2404.19315/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions](https://mlsanity.com/paper/1902.00508/) (2019)
- [Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise](https://mlsanity.com/paper/2109.06772/) (2021)
- [Machine Translation of Low-Resource Spoken Dialects: Strategies for Normalizing Swiss German](https://mlsanity.com/paper/1710.11035/) (2017)
- [BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text](https://mlsanity.com/paper/2311.00116/) (2023)
- [Exploring Word Embeddings for Unsupervised Textual User-Generated Content Normalization](https://mlsanity.com/paper/1704.02963/) (2017)

## Built on by (most cited first, 2 of 2)

- [Modern Models, Medieval Texts: A POS Tagging Study of Old Occitan](https://mlsanity.com/paper/2503.07827/) (2025)
- [Historical German Text Normalization Using Type- and Token-Based Language Modeling](https://mlsanity.com/paper/2409.02841/) (2024)

## Abstract (arXiv)

Effectively normalizing textual data poses a considerable challenge, especially for low-resource languages lacking standardized writing systems. In this study, we fine-tuned a multilingual model with data from several Occitan dialects and conducted a series of experiments to assess the model's representations of these dialects. For evaluation purposes, we compiled a parallel lexicon encompassing four Occitan dialects. Intrinsic evaluations of the model's embeddings revealed that surface similarity between the dialects strengthened representations. When the model was further fine-tuned for part-of-speech tagging and Universal Dependency parsing, its performance was robust to dialectical variation, even when trained solely on part-of-speech data from a single dialect. Our findings suggest that large multilingual models minimize the need for spelling normalization during pre-processing.