mlsanityData to 14 Oct 2026

NoCoLA: The Norwegian Corpus of Linguistic Acceptability

arXiv 2306.07790 ↗First posted 13 Jun 2023cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

While there has been a surge of large language models for Norwegian in recent years, we lack any tool to evaluate their understanding of grammaticality. We present two new Norwegian datasets for this task. NoCoLA_class is a supervised binary classification task where the goal is to discriminate between acceptable and non-acceptable sentences. On the other hand, NoCoLA_zero is a purely diagnostic task for evaluating the grammatical judgement of a language model in a completely zero-shot manner, i.e. without any further training. In this paper, we describe both datasets in detail, show how to use them for different flavors of language models, and conduct a comparative study of the existing Norwegian language models.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

17 of 34 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  8. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  9. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  10. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  11. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  12. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
5 more
  1. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  2. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  3. 2021Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpuscited by 18
  4. 2021DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharingcited by 16
  5. 2022Geographic Adaptation of Pretrained Language Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 1.

10 papers.

  1. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  2. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  3. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  4. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 8
  5. 2025Minimal Pair-Based Evaluation of Code-Switchingcited by 8
  6. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 7
  7. 2025DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errorscited by 3
  8. 2024Quantum Transfer Learning for Acceptability Judgementscited by 2
  9. 2025COLA-GEC: A Bidirectional Framework for Enhancing Grammatical Acceptability and Error Correctioncited by 0
  10. 2026How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarkingcited by 0