# Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect

arXiv: https://arxiv.org/abs/2401.14400 · HTML: https://mlsanity.com/paper/2401.14400/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 14 of 14)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [Improving Pre-Trained Multilingual Models with Vocabulary Expansion](https://mlsanity.com/paper/1909.12440/) (2019)
- [Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise](https://mlsanity.com/paper/2109.06772/) (2021)
- [A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography](https://mlsanity.com/paper/2311.16865/) (2023)
- [SwissBERT: The Multilingual Language Model for Switzerland](https://mlsanity.com/paper/2303.13310/) (2023)
- [What is the best recipe for character-level encoder-only modelling?](https://mlsanity.com/paper/2305.05461/) (2023)
- [Automatic Creation of Text Corpora for Low-Resource Languages from the Internet: The Case of Swiss German](https://mlsanity.com/paper/1912.00159/) (2019)

## Built on by (most cited first, 2 of 2)

- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapters](https://mlsanity.com/paper/2501.06025/) (2025)

## Abstract (arXiv)

Creating neural text encoders for written Swiss German is challenging due to a dearth of training data combined with dialectal variation. In this paper, we build on several existing multilingual encoders and adapt them to Swiss German using continued pre-training. Evaluation on three diverse downstream tasks shows that simply adding a Swiss German adapter to a modular encoder achieves 97.5% of fully monolithic adaptation performance. We further find that for the task of retrieving Swiss German sentences given Standard German queries, adapting a character-level model is more effective than the other adaptation strategies. We release our code and the models trained for our experiments at https://github.com/ZurichNLP/swiss-german-text-encoders