# NoCoLA: The Norwegian Corpus of Linguistic Acceptability

arXiv: https://arxiv.org/abs/2306.07790 · HTML: https://mlsanity.com/paper/2306.07790/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [GLGE: A New General Language Generation Evaluation Benchmark](https://mlsanity.com/paper/2011.11928/) (2020)
- [RuCoLA: Russian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2210.12814/) (2022)
- [Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model](https://mlsanity.com/paper/2104.09617/) (2021)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus](https://mlsanity.com/paper/2109.12053/) (2021)
- [DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing](https://mlsanity.com/paper/2105.06681/) (2021)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)

## Built on by (most cited first, 10 of 10)

- [RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2406.19232/) (2024)
- [MELA: Multilingual Evaluation of Linguistic Acceptability](https://mlsanity.com/paper/2311.09033/) (2023)
- [Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2411.07474/) (2024)
- [NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark](https://mlsanity.com/paper/2504.07749/) (2025)
- [Minimal Pair-Based Evaluation of Code-Switching](https://mlsanity.com/paper/2506.01840/) (2025)
- [HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models](https://mlsanity.com/paper/2511.01066/) (2025)
- [DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors](https://mlsanity.com/paper/2512.04799/) (2025)
- [Quantum Transfer Learning for Acceptability Judgements](https://mlsanity.com/paper/2401.07777/) (2024)
- [COLA-GEC: A Bidirectional Framework for Enhancing Grammatical Acceptability and Error Correction](https://mlsanity.com/paper/2507.11867/) (2025)
- [How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarking](https://mlsanity.com/paper/2610.19185/) (2026)

## Abstract (arXiv)

While there has been a surge of large language models for Norwegian in recent years, we lack any tool to evaluate their understanding of grammaticality. We present two new Norwegian datasets for this task. NoCoLA_class is a supervised binary classification task where the goal is to discriminate between acceptable and non-acceptable sentences. On the other hand, NoCoLA_zero is a purely diagnostic task for evaluating the grammatical judgement of a language model in a completely zero-shot manner, i.e. without any further training. In this paper, we describe both datasets in detail, show how to use them for different flavors of language models, and conduct a comparative study of the existing Norwegian language models.