# NorBench -- A Benchmark for Norwegian Language Models

arXiv: https://arxiv.org/abs/2305.03880 · HTML: https://mlsanity.com/paper/2305.03880/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 6 May 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [NormFormer: Improved Transformer Pretraining with Extra Normalization](https://mlsanity.com/paper/2110.09456/) (2021)
- [GLGE: A New General Language Generation Evaluation Benchmark](https://mlsanity.com/paper/2011.11928/) (2020)
- [Trained on 100 million words and still in shape: BERT meets British National Corpus](https://mlsanity.com/paper/2303.09859/) (2023)
- [A Fine-Grained Sentiment Dataset for Norwegian](https://mlsanity.com/paper/1911.12722/) (2019)
- [Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model](https://mlsanity.com/paper/2104.09617/) (2021)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [NorNE: Annotating Named Entities for Norwegian](https://mlsanity.com/paper/1911.12146/) (2019)
- [NoReC: The Norwegian Review Corpus](https://mlsanity.com/paper/1710.05370/) (2017)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [Measuring Harmful Representations in Scandinavian Language Models](https://mlsanity.com/paper/2211.11678/) (2022)
- [NorQuAD: Norwegian Question Answering Dataset](https://mlsanity.com/paper/2305.01957/) (2023)
- [Entity-Level Sentiment Analysis (ELSA): An exploratory task survey](https://mlsanity.com/paper/2304.14241/) (2023)
- [Measuring Normative and Descriptive Biases in Language Models Using Census Data](https://mlsanity.com/paper/2304.05764/) (2023)

## Built on by (most cited first, 23 of 23)

- [Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases](https://mlsanity.com/paper/2502.19249/) (2025)
- [Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks](https://mlsanity.com/paper/2406.13469/) (2024)
- [The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding](https://mlsanity.com/paper/2406.02396/) (2024)
- [Small Languages, Big Models: A Study of Continual Training on Languages of Norway](https://mlsanity.com/paper/2412.06484/) (2024)
- [NoCoLA: The Norwegian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2306.07790/) (2023)
- [NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian](https://mlsanity.com/paper/2312.01314/) (2023)
- [The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective](https://mlsanity.com/paper/2412.09460/) (2024)
- [NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark](https://mlsanity.com/paper/2504.07749/) (2025)
- [HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models](https://mlsanity.com/paper/2511.01066/) (2025)
- [It's Difficult to be Neutral -- Human and LLM-based Sentiment Annotation of Patient Comments](https://mlsanity.com/paper/2404.18832/) (2024)
- [Exploring Multiple Strategies to Improve Multilingual Coreference Resolution in CorefUD](https://mlsanity.com/paper/2408.16893/) (2024)
- [skLEP: A Slovak General Language Understanding Benchmark](https://mlsanity.com/paper/2506.21508/) (2025)
- [Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detection](https://mlsanity.com/paper/2501.03870/) (2025)
- [Multi-label Scandinavian Language Identification (SLIDE)](https://mlsanity.com/paper/2502.06692/) (2025)
- [Entity-Level Sentiment: More than the Sum of Its Parts](https://mlsanity.com/paper/2407.03916/) (2024)
- [HiTZ at VarDial 2025 NorSID: Overcoming Data Scarcity with Language Transfer and Automatic Data Annotation](https://mlsanity.com/paper/2412.10095/) (2024)
- [Mixed Feelings: Cross-Domain Sentiment Classification of Patient Feedback](https://mlsanity.com/paper/2501.19134/) (2025)
- [When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels](https://mlsanity.com/paper/2605.06652/) (2026)
- [Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings](https://mlsanity.com/paper/2310.19420/) (2023)
- [NorwAI's Large Language Models: Technical Report](https://mlsanity.com/paper/2601.03034/) (2026)
- [Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite](https://mlsanity.com/paper/2604.01957/) (2026)
- [KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts](https://mlsanity.com/paper/2606.01904/) (2026)
- [SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models](https://mlsanity.com/paper/2608.06179/) (2026)

## Abstract (arXiv)

We present NorBench: a streamlined suite of NLP tasks and probes for evaluating Norwegian language models (LMs) on standardized data splits and evaluation metrics. We also introduce a range of new Norwegian language models (both encoder and encoder-decoder based). Finally, we compare and analyze their performance, along with other existing LMs, across the different benchmark tests of NorBench.