# WikiBERT models: deep transfer learning for many languages

arXiv: https://arxiv.org/abs/2006.01538 · HTML: https://mlsanity.com/paper/2006.01538/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [FinGPT: Large Generative Models for a Small Language](https://mlsanity.com/paper/2311.05640/) (2023)
- [When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages](https://mlsanity.com/paper/2311.09205/) (2023)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan](https://mlsanity.com/paper/2107.07903/) (2021)
- [Predicting the Performance of Multilingual NLP Models](https://mlsanity.com/paper/2110.08875/) (2021)
- [Poro 34B and the Blessing of Multilinguality](https://mlsanity.com/paper/2404.01856/) (2024)
- [gaBERT -- an Irish Language Model](https://mlsanity.com/paper/2107.12930/) (2021)
- [SlovakBERT: Slovak Masked Language Model](https://mlsanity.com/paper/2109.15254/) (2021)
- [VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding](https://mlsanity.com/paper/2403.15882/) (2024)
- [RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning](https://mlsanity.com/paper/2102.11278/) (2021)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models](https://mlsanity.com/paper/2603.29552/) (2026)
- [bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2306.02349/) (2023)
- [ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization](https://mlsanity.com/paper/2109.04098/) (2021)
- [Bilingual Language Modeling, A transfer learning technique for Roman Urdu](https://mlsanity.com/paper/2102.10958/) (2021)
- [Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension](https://mlsanity.com/paper/2303.13355/) (2023)
- [AlcLaM: Arabic Dialectal Language Model](https://mlsanity.com/paper/2407.13097/) (2024)
- [Approaches to Semantic Textual Similarity in Slovak Language: From Algorithms to Transformers](https://mlsanity.com/paper/2602.04659/) (2026)
- [YoNER: A New Yor`ub'a Multi-domain Named Entity Recognition Dataset](https://mlsanity.com/paper/2604.05624/) (2026)
- [Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining](https://mlsanity.com/paper/2609.21362/) (2026)

## Abstract (arXiv)

Deep neural language models such as BERT have enabled substantial recent advances in many natural language processing tasks. Due to the effort and computational cost involved in their pre-training, language-specific models are typically introduced only for a small number of high-resource languages such as English. While multilingual models covering large numbers of languages are available, recent work suggests monolingual training can produce better models, and our understanding of the tradeoffs between mono- and multilingual training is incomplete. In this paper, we introduce a simple, fully automated pipeline for creating language-specific BERT models from Wikipedia data and introduce 42 new such models, most for languages up to now lacking dedicated deep neural language models. We assess the merits of these models using the state-of-the-art UDify parser on Universal Dependencies data, contrasting performance with results using the multilingual BERT model. We find that UDify using WikiBERT models outperforms the parser using mBERT on average, with the language-specific models showing substantially improved performance for some languages, yet limited improvement or a decrease in performance for others. We also present preliminary results as first steps toward an understanding of the conditions under which language-specific models are most beneficial. All of the methods and models introduced in this work are available under open licenses from https://github.com/turkunlp/wikibert.