# Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model

arXiv: https://arxiv.org/abs/2104.09617 · HTML: https://mlsanity.com/paper/2104.09617/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Playing with Words at the National Library of Sweden -- Making a Swedish BERT](https://mlsanity.com/paper/2007.01658/) (2020)
- [A Fine-Grained Sentiment Dataset for Norwegian](https://mlsanity.com/paper/1911.12722/) (2019)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [NorNE: Annotating Named Entities for Norwegian](https://mlsanity.com/paper/1911.12146/) (2019)
- [A Finnish News Corpus for Named Entity Recognition](https://mlsanity.com/paper/1908.04212/) (2019)

## Built on by (most cited first, 23 of 23)

- [The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding](https://mlsanity.com/paper/2406.02396/) (2024)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [Small Languages, Big Models: A Study of Continual Training on Languages of Norway](https://mlsanity.com/paper/2412.06484/) (2024)
- [NoCoLA: The Norwegian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2306.07790/) (2023)
- [NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian](https://mlsanity.com/paper/2312.01314/) (2023)
- [GPT-SW3: An Autoregressive Language Model for the Nordic Languages](https://mlsanity.com/paper/2305.12987/) (2023)
- [The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective](https://mlsanity.com/paper/2412.09460/) (2024)
- [Measuring Harmful Representations in Scandinavian Language Models](https://mlsanity.com/paper/2211.11678/) (2022)
- [NorQuAD: Norwegian Question Answering Dataset](https://mlsanity.com/paper/2305.01957/) (2023)
- [Training a T5 Using Lab-sized Resources](https://mlsanity.com/paper/2208.12097/) (2022)
- [Whispering in Norwegian: Navigating Orthographic and Dialectic Challenges](https://mlsanity.com/paper/2402.01917/) (2024)
- [Measuring Normative and Descriptive Biases in Language Models Using Census Data](https://mlsanity.com/paper/2304.05764/) (2023)
- [Annotating Norwegian Language Varieties on Twitter for Part-of-Speech](https://mlsanity.com/paper/2210.06150/) (2022)
- [Boosting Norwegian Automatic Speech Recognition](https://mlsanity.com/paper/2307.01672/) (2023)
- [QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments](https://mlsanity.com/paper/2508.16867/) (2025)
- [English K_Quantization of LLMs Does Not Disproportionately Diminish Multilingual Performance](https://mlsanity.com/paper/2503.03592/) (2025)
- [Aligning the Norwegian UD Treebank with Entity and Coreference Information](https://mlsanity.com/paper/2305.13527/) (2023)
- [A Look Into News Avoidance Through AWRS: An Avoidance-Aware Recommender System](https://mlsanity.com/paper/2407.09137/) (2024)
- [Data Processing for the OpenGPT-X Model Family](https://mlsanity.com/paper/2410.08800/) (2024)
- [NorwAI's Large Language Models: Technical Report](https://mlsanity.com/paper/2601.03034/) (2026)
- [Rethinking Metrics for Lexical Semantic Change Detection](https://mlsanity.com/paper/2602.15716/) (2026)
- [MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation](https://mlsanity.com/paper/2602.21379/) (2026)

## Abstract (arXiv)

In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmaal and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.