# Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise arXiv: https://arxiv.org/abs/2109.06772 ยท HTML: https://mlsanity.com/paper/2109.06772/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 42 references that are in the map. - First posted on arXiv 14 Sep 2021. ## Builds on (most cited first, 21 of 21) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019) - [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019) - [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020) - [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019) - [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015) - [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017) - [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019) - [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020) - [Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation](https://mlsanity.com/paper/1902.01509/) (2019) - [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021) - [Improving Robustness of Machine Translation with Synthetic Noise](https://mlsanity.com/paper/1902.09508/) (2019) - [Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1909.00437/) (2019) - [How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?](https://mlsanity.com/paper/1704.04441/) (2017) - [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021) - [Robsut Wrod Reocginiton via semi-Character Recurrent Neural Network](https://mlsanity.com/paper/1608.02214/) (2016) - [MOROCO: The Moldavian and Romanian Dialectal Corpus](https://mlsanity.com/paper/1901.06543/) (2019) - [Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi](https://mlsanity.com/paper/2005.00318/) (2020) - [Neural Machine Translation of Text from Non-Native Speakers](https://mlsanity.com/paper/1808.06267/) (2018) - [Automatic Creation of Text Corpora for Low-Resource Languages from the Internet: The Case of Swiss German](https://mlsanity.com/paper/1912.00159/) (2019) ## Built on by (most cited first, 19 of 19) - [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024) - [Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages](https://mlsanity.com/paper/2304.10158/) (2023) - [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023) - [Evaluating Pixel Language Models on Non-Standardized Languages](https://mlsanity.com/paper/2412.09084/) (2024) - [Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages](https://mlsanity.com/paper/2303.17683/) (2023) - [MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank](https://mlsanity.com/paper/2403.10293/) (2024) - [BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text](https://mlsanity.com/paper/2311.00116/) (2023) - [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024) - [Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect](https://mlsanity.com/paper/2401.14400/) (2024) - [Modeling Orthographic Variation in Occitan's Dialects](https://mlsanity.com/paper/2404.19315/) (2024) - [Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detection](https://mlsanity.com/paper/2501.03870/) (2025) - [Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants](https://mlsanity.com/paper/2506.15239/) (2025) - [CharSpan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages](https://mlsanity.com/paper/2305.05214/) (2023) - [Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages](https://mlsanity.com/paper/2506.17715/) (2025) - [Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarios](https://mlsanity.com/paper/2501.11269/) (2025) - [Benchmarking NLP-supported Language Sample Analysis for Swiss Children's Speech](https://mlsanity.com/paper/2504.00780/) (2025) - [textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language](https://mlsanity.com/paper/2603.28213/) (2026) - [Small Data, Big Noise: Adversarial Training for Robust Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2606.10610/) (2026) - [Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustness](https://mlsanity.com/paper/2608.05510/) (2026) ## Abstract (arXiv) Cross-lingual transfer between a high-resource language and its dialects or closely related language varieties should be facilitated by their similarity. However, current approaches that operate in the embedding space do not take surface similarity into account. This work presents a simple yet effective strategy to imrove cross-lingual transfer between closely related varieties. We propose to augment the data of the high-resource source language with character-level noise to make the model more robust towards spelling variations. Our strategy shows consistent improvements over several languages and tasks: Zero-shot transfer of POS tagging and topic identification between language varieties from the Finnic, West and North Germanic, and Western Romance language branches. Our work provides evidence for the usefulness of simple surface-level noise in improving transfer between language varieties.