mlsanityData to 14 Oct 2026

DUMB: A Benchmark for Smart Evaluation of Dutch Models

arXiv 2305.13026 ↗First posted 22 May 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce the Dutch Model Benchmark: DUMB. The benchmark includes a diverse set of datasets for low-, medium- and high-resource tasks. The total set of nine tasks includes four tasks that were previously not available in Dutch. Instead of relying on a mean score across tasks, we propose Relative Error Reduction (RER), which compares the DUMB performance of language models to a strong baseline which can be referred to in the future even when assessing different sets of language models. Through a comparison of 14 pre-trained language models (mono- and multi-lingual, of varying sizes), we assess the internal consistency of the benchmark tasks, as well as the factors that likely enable high performance. Our results indicate that current Dutch monolingual models under-perform and suggest training larger Dutch models with other architectures and pre-training objectives. At present, the highest performance is achieved by DeBERTaV3 (large), XLM-R (large) and mDeBERTaV3 (base). In addition to highlighting best strategies for training larger Dutch models, DUMB will foster further research on Dutch. A public leaderboard is available at https://dumbench.nl.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 39 of its 73 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

39 of 73 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
27 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  2. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  3. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  4. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  5. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  6. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  7. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  8. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  9. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  10. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  11. 2019BERTje: A Dutch BERT Modelcited by 109
  12. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  13. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  14. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  15. 2021Towards Zero-Label Language Learningcited by 76
  16. 2020Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Societycited by 72
  17. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  18. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  19. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47
  20. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
  21. 2021SICKNL: A Dataset for Dutch Natural Language Inferencecited by 20
  22. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  23. 2019The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviewscited by 14
  24. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  25. 2021Can BERT Dig It? -- Named Entity Recognition for Information Retrieval in the Archaeology Domaincited by 5
  26. 2021Cross-lingual Transferring of Pre-trained Contextualized Language Modelscited by 3
  27. 2022RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Usecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 2.

6 papers.

  1. 2023Language Resources for Dutch Large Language Modellingcited by 11
  2. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 5
  3. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutchcited by 5
  4. 2025skLEP: A Slovak General Language Understanding Benchmarkcited by 3
  5. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  6. 2026From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutchcited by 0