mlsanityData to 7 Oct 2026

ScandEval: A Benchmark for Scandinavian Natural Language Processing

arXiv 2304.00906 ↗First posted 3 Apr 2023cs.CL cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper introduces a Scandinavian benchmarking platform, ScandEval, which can benchmark any pretrained model on four different tasks in the Scandinavian languages. The datasets used in two of the tasks, linguistic acceptability and question answering, are new. We develop and release a Python package and command-line interface, scandeval, which can benchmark any model that has been uploaded to the Hugging Face Hub, with reproducible results. Using this package, we benchmark more than 100 Scandinavian or multilingual models and present the results of these in an interactive online leaderboard, as well as provide an analysis of the results. The analysis shows that there is substantial cross-lingual transfer among the Mainland Scandinavian languages (Danish, Swedish and Norwegian), with limited cross-lingual transfer between the group of Mainland Scandinavian languages and the group of Insular Scandinavian languages (Icelandic and Faroese). The benchmarking results also show that the investment in language technology in Norway, Sweden and Denmark has led to language models that outperform massively multilingual models such as XLM-RoBERTa and mDeBERTaV3. We release the source code for both the package and leaderboard.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 68 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

29 of 68 references.

References · 29

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  6. 2018Neural Network Acceptability Judgmentscited by 952
  7. 2019How multilingual is Multilingual BERT?cited by 642
  8. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  10. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  11. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  12. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
17 more
  1. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  2. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  3. 2020KLEJ: Comprehensive Benchmark for Polish Language Understandingcited by 51
  4. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 44
  5. 2020ParsiNLU: A Suite of Language Understanding Challenges for Persiancited by 42
  6. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
  7. 2020Playing with Words at the National Library of Sweden -- Making a Swedish BERTcited by 35
  8. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32
  9. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  10. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  11. 2019NorNE: Annotating Named Entities for Norwegiancited by 18
  12. 2017NoReC: The Norwegian Review Corpuscited by 17
  13. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
  14. 2022This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polishcited by 13
  15. 2021DaCy: A Unified Framework for Danish NLPcited by 10
  16. 2022Mukayese: Turkish NLP Strikes Backcited by 9
  17. 2020Why Not Simply Translate? A First Swedish Evaluation Benchmark for Semantic Similaritycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 6; 2026: 2.

16 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  2. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  3. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  4. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  5. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 10
  6. 2024GEITje 7B Ultra: A Conversational Model for Dutchcited by 4
  7. 2026Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedishcited by 3
  8. 2024Evaluating Large Language Models with Human Feedback: Establishing a Swedish Benchmarkcited by 1
  9. 2024SWEb: A Large Web Dataset for the Scandinavian Languagescited by 1
  10. 2025Bielik 11B v2 Technical Reportcited by 1
  11. 2023Augmenty: A Python Library for Structured Text Augmentationcited by 0
  12. 2024DANSK and DaCy 2.6.0: Domain Generalization of Danish Named Entity Recognitioncited by 0
4 more
  1. 2024Exposing Assumptions in AI Benchmarks through Cognitive Modellingcited by 0
  2. 2025Natural Language Processing for Electronic Health Records in Scandinavian Languages: Norwegian, Swedish, and Danishcited by 0
  3. 2025COLA-GEC: A Bidirectional Framework for Enhancing Grammatical Acceptability and Error Correctioncited by 0
  4. 2025Automated HIV Screening on Dutch Electronic Health Records with Large Language Modelscited by 0