mlsanityData to 7 Oct 2026

BERTje: A Dutch BERT Model

arXiv 1912.09582 ↗First posted 19 Dec 2019cs.CLcited by 109

Not on the map. Why?

Abstract (arXiv)

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 22; 2023: 19; 2024: 16; 2025: 10; 2026: 6.

109 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  5. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  6. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  7. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  8. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  9. 2021Pre-Training BERT on Arabic Tweets: Practical Considerationscited by 39
  10. 2021MarIA: Spanish Language Modelscited by 36
  11. 2021The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primatescited by 32
  12. 2021Multilingual Language Models Predict Human Reading Behaviorcited by 29
38 more
  1. 2021A systematic review of Hate Speech automatic detection using Natural Language Processingcited by 26
  2. 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26
  3. 2021DATE: Detecting Anomalies in Text via Self-Supervision of Transformerscited by 25
  4. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  5. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  6. 2023Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*cited by 24
  7. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  8. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  9. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  10. 2021BERTi'c -- The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbiancited by 21
  11. 2021SICKNL: A Dataset for Dutch Natural Language Inferencecited by 20
  12. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  13. 2023Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languagescited by 19
  14. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  15. 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
  16. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  17. 2023Do large language models solve verbal analogies like children do?cited by 13
  18. 2021Adapting Monolingual Models: Data can be Scarce when Language Similarity is Highcited by 12
  19. 2021gaBERT -- an Irish Language Modelcited by 12
  20. 2023Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptationcited by 11
  21. 2024Exploration of Attention Mechanism-Enhanced Deep Learning Models in the Mining of Medical Textual Datacited by 11
  22. 2021KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understandingcited by 9
  23. 2022Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studiescited by 9
  24. 2022Lessons learned from the evaluation of Spanish Language Modelscited by 8
  25. 2021Are the Multilingual Models Better? Improving Czech Sentiment with Transformerscited by 7
  26. 2021PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptorscited by 6
  27. 2022ALBETO and DistilBETO: Lightweight Spanish Language Modelscited by 6
  28. 2023Can Peanuts Fall in Love with Distributional Semantics?cited by 6
  29. 2023Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languagescited by 6
  30. 2023DUMB: A Benchmark for Smart Evaluation of Dutch Modelscited by 6
  31. 2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6
  32. 2021Can BERT Dig It? -- Named Entity Recognition for Information Retrieval in the Archaeology Domaincited by 5
  33. 2024Towards Tailored Recovery of Lexical Diversity in Literary Machine Translationcited by 5
  34. 2024ChocoLlama: Lessons Learned From Teaching Llamas Dutchcited by 5
  35. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutchcited by 5
  36. 2021TMR: Evaluating NER Recall on Tough Mentionscited by 4
  37. 2021Improving BERT Pretraining with Syntactic Supervisioncited by 4
  38. 2022RobBERTje: a Distilled Dutch BERT Modelcited by 4

50 most cited of 109 All 109 (JSON)