mlsanityData to 7 Oct 2026

Playing with Words at the National Library of Sweden -- Making a Swedish BERT

arXiv 2007.01658 ↗First posted 3 Jul 2020cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper introduces the Swedish BERT ("KB-BERT") developed by the KBLab for data-driven research at the National Library of Sweden (KB). Building on recent efforts to create transformer-based BERT models for languages other than English, we explain how we used KB's collections to create and train a new language-specific BERT model for Swedish. We also present the results of our model in comparison with existing models - chiefly that produced by the Swedish Public Employment Service, Arbetsf"ormedlingen, and Google's multilingual M-BERT - where we demonstrate that KB-BERT outperforms these in a range of NLP tasks from named entity recognition (NER) to part-of-speech tagging (POS). Our discussion highlights the difficulties that continue to exist given the lack of training data and testbeds for smaller languages like Swedish. We release our model for further exploration and research here: https://github.com/Kungbib/swedish-bert-models .

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 7; 2023: 8; 2024: 3; 2025: 5; 2026: 2.

35 papers.

Sort:

  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  3. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  4. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  5. 2022Acceptability Judgements via Examining the Topology of Attention Mapscited by 19
  6. 2023ScandEval: A Benchmark for Scandinavian Natural Language Processingcited by 16
  7. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  8. 2021BERT-based distractor generation for Swedish reading comprehension questions using a small-scale datasetcited by 12
  9. 2021Training dataset and dictionary sizes matter in BERT models: the case of Baltic languagescited by 7
  10. 2021PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptorscited by 6
  11. 2023Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languagescited by 6
  12. 2024A Survey of Large Language Models for European Languagescited by 6
23 more
  1. 2021Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registerscited by 5
  2. 2021Evaluation of contextual embeddings on less-resourced languagescited by 4
  3. 2023Distributional Data Augmentation Methods for Low Resource Languagecited by 4
  4. 2022Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognitioncited by 3
  5. 2023SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedishcited by 3
  6. 2021Enhanced Universal Dependency Parsing with Automated Concatenation of Embeddingscited by 2
  7. 2022Hearing voices at the National Library -- a speech corpus and acoustic model for the Swedish languagecited by 2
  8. 2022Sort by Structure: Language Model Ranking as Dependency Probingcited by 2
  9. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  10. 2023Bipol: Multi-axes Evaluation of Bias with Explainability in Benchmark Datasetscited by 2
  11. 2025QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgmentscited by 2
  12. 2022Local Structure Matters Most in Most Languagescited by 1
  13. 2025Data-Constrained Synthesis of Training Data for De-Identificationcited by 1
  14. 2025Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimationcited by 1
  15. 2025Matching and Linking Entries in Historical Swedish Encyclopediascited by 1
  16. 2022Punctuation restoration in Swedish through fine-tuned KB-BERTcited by 0
  17. 2023Annotated Job Ads with Named Entity Recognitioncited by 0
  18. 2023Text Annotation Handbook: A Practical Guide for Machine Learning Projectscited by 0
  19. 2024Mapping the Past: Geographically Linking an Early 20th Century Swedish Encyclopedia with Wikidatacited by 0
  20. 2024AlcLaM: Arabic Dialectal Language Modelcited by 0
  21. 2025The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classificationcited by 0
  22. 2026Rethinking Metrics for Lexical Semantic Change Detectioncited by 0
  23. 2026ATLAS: Article Tracking, Linking, and Analysis of Swedish Encyclopediascited by 0