mlsanityData to 7 Oct 2026

NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark

arXiv 2504.07749 ↗First posted 10 Apr 2025cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets -- of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmaal and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

45 of 68 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
33 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  7. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  8. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  9. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  11. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
  12. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  13. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  14. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  15. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  16. 2019Joint Reasoning for Temporal and Causal Relationscited by 60
  17. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  18. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  19. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  20. 2019A Fine-Grained Sentiment Dataset for Norwegiancited by 23
  21. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  22. 2019NorNE: Annotating Named Entities for Norwegiancited by 18
  23. 2017NoReC: The Norwegian Review Corpuscited by 17
  24. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  25. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 15
  26. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  27. 2023NoCoLA: The Norwegian Corpus of Linguistic Acceptabilitycited by 9
  28. 2023NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegiancited by 9
  29. 2024The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspectivecited by 8
  30. 2024HREF: Human Response-Guided Evaluation of Instruction Following in Language Modelscited by 8
  31. 2023NorQuAD: Norwegian Question Answering Datasetcited by 7
  32. 2025Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articlescited by 2
  33. 2025A Collection of Question Answering Datasets for Norwegiancited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 6
  2. 2026Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedishcited by 3
  3. 2026When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labelscited by 1
  4. 2025Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languagescited by 0
  5. 2025FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Modelscited by 0
  6. 2026NorwAI's Large Language Models: Technical Reportcited by 0
  7. 2026MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languagescited by 0