mlsanityData to 7 Oct 2026

SwissBERT: The Multilingual Language Model for Switzerland

arXiv 2303.13310 ↗First posted 23 Mar 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

We present SwissBERT, a masked language model created specifically for processing Switzerland-related text. SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland -- German, French, Italian, and Romansh. We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work. The model and our open-source code are publicly released at https://github.com/ZurichNLP/swissbert.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

27 of 51 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  12. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
15 more
  1. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  2. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  3. 2019CamemBERT: a Tasty French Language Modelcited by 368
  4. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  5. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  6. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  7. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  8. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  9. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  10. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  11. 2020German's Next Language Modelcited by 144
  12. 2020GottBERT: a pure German Language Modelcited by 49
  13. 2020X-Stance: A Multilingual Multi-Target Dataset for Stance Detectioncited by 42
  14. 2022hmBERT: Historical Multilingual Language Models for Named Entity Recognitioncited by 8
  15. 2021MDAPT: Multilingual Domain Adaptive Pretraining in a Single Modelcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 1.

8 papers.

  1. 2024Fine-Grained Prediction of Reading Comprehension from Eye Movementscited by 9
  2. 2023Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Modelscited by 7
  3. 2025The Mediomatix Corpus: Parallel Data for Romansh Language Varieties via Comparable Schoolbookscited by 3
  4. 2024Modular Adaptation of Multilingual Encoders to Written Swiss German Dialectcited by 2
  5. 2024Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documentscited by 2
  6. 2026Robust Language Identification for Romansh Varietiescited by 2
  7. 2024From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudencecited by 1
  8. 2025CodeNER: Code Prompting for Named Entity Recognitioncited by 0