# SwissBERT: The Multilingual Language Model for Switzerland

arXiv: https://arxiv.org/abs/2303.13310 · HTML: https://mlsanity.com/paper/2303.13310/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [X-Stance: A Multilingual Multi-Target Dataset for Stance Detection](https://mlsanity.com/paper/2003.08385/) (2020)
- [hmBERT: Historical Multilingual Language Models for Named Entity Recognition](https://mlsanity.com/paper/2205.15575/) (2022)
- [MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model](https://mlsanity.com/paper/2109.06605/) (2021)

## Built on by (most cited first, 9 of 9)

- [Fine-Grained Prediction of Reading Comprehension from Eye Movements](https://mlsanity.com/paper/2410.04484/) (2024)
- [Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models](https://mlsanity.com/paper/2308.11103/) (2023)
- [The Mediomatix Corpus: Parallel Data for Romansh Language Varieties via Comparable Schoolbooks](https://mlsanity.com/paper/2508.16371/) (2025)
- [Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect](https://mlsanity.com/paper/2401.14400/) (2024)
- [Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documents](https://mlsanity.com/paper/2405.07513/) (2024)
- [Robust Language Identification for Romansh Varieties](https://mlsanity.com/paper/2603.15969/) (2026)
- [From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudence](https://mlsanity.com/paper/2410.13460/) (2024)
- [CodeNER: Code Prompting for Named Entity Recognition](https://mlsanity.com/paper/2507.20423/) (2025)
- [Swiss4Arg: A Parallel Dataset for Multilingual Argument Mining across Swiss National Languages](https://mlsanity.com/paper/2610.17894/) (2026)

## Abstract (arXiv)

We present SwissBERT, a masked language model created specifically for processing Switzerland-related text. SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland -- German, French, Italian, and Romansh. We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work. The model and our open-source code are publicly released at https://github.com/ZurichNLP/swissbert.