# Playing with Words at the National Library of Sweden -- Making a Swedish BERT

arXiv: https://arxiv.org/abs/2007.01658 · HTML: https://mlsanity.com/paper/2007.01658/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 30 of 35)

- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model](https://mlsanity.com/paper/2104.09617/) (2021)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [Acceptability Judgements via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2205.09630/) (2022)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [Cross-lingual Transfer of Monolingual Models](https://mlsanity.com/paper/2109.07348/) (2021)
- [BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset](https://mlsanity.com/paper/2108.03973/) (2021)
- [Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages](https://mlsanity.com/paper/2112.10553/) (2021)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers](https://mlsanity.com/paper/2102.07396/) (2021)
- [Evaluation of contextual embeddings on less-resourced languages](https://mlsanity.com/paper/2107.10614/) (2021)
- [Distributional Data Augmentation Methods for Low Resource Language](https://mlsanity.com/paper/2309.04862/) (2023)
- [Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition](https://mlsanity.com/paper/2202.02617/) (2022)
- [Hearing voices at the National Library -- a speech corpus and acoustic model for the Swedish language](https://mlsanity.com/paper/2205.03026/) (2022)
- [SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish](https://mlsanity.com/paper/2304.13994/) (2023)
- [Enhanced Universal Dependency Parsing with Automated Concatenation of Embeddings](https://mlsanity.com/paper/2107.02416/) (2021)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes](https://mlsanity.com/paper/2211.05015/) (2022)
- [Bipol: Multi-axes Evaluation of Bias with Explainability in Benchmark Datasets](https://mlsanity.com/paper/2301.12139/) (2023)
- [QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments](https://mlsanity.com/paper/2508.16867/) (2025)
- [Local Structure Matters Most in Most Languages](https://mlsanity.com/paper/2211.05025/) (2022)
- [Data-Constrained Synthesis of Training Data for De-Identification](https://mlsanity.com/paper/2502.14677/) (2025)
- [Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimation](https://mlsanity.com/paper/2505.04643/) (2025)
- [Matching and Linking Entries in Historical Swedish Encyclopedias](https://mlsanity.com/paper/2507.01170/) (2025)
- [Punctuation restoration in Swedish through fine-tuned KB-BERT](https://mlsanity.com/paper/2202.06769/) (2022)
- [Annotated Job Ads with Named Entity Recognition](https://mlsanity.com/paper/2310.11769/) (2023)
- [Text Annotation Handbook: A Practical Guide for Machine Learning Projects](https://mlsanity.com/paper/2310.11780/) (2023)

## Abstract (arXiv)

This paper introduces the Swedish BERT ("KB-BERT") developed by the KBLab for data-driven research at the National Library of Sweden (KB). Building on recent efforts to create transformer-based BERT models for languages other than English, we explain how we used KB's collections to create and train a new language-specific BERT model for Swedish. We also present the results of our model in comparison with existing models - chiefly that produced by the Swedish Public Employment Service, Arbetsf"ormedlingen, and Google's multilingual M-BERT - where we demonstrate that KB-BERT outperforms these in a range of NLP tasks from named entity recognition (NER) to part-of-speech tagging (POS). Our discussion highlights the difficulties that continue to exist given the lack of training data and testbeds for smaller languages like Swedish. We release our model for further exploration and research here: https://github.com/Kungbib/swedish-bert-models .