# EstBERT: A Pretrained Language-Specific BERT for Estonian

arXiv: https://arxiv.org/abs/2011.04784 · HTML: https://mlsanity.com/paper/2011.04784/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media](https://mlsanity.com/paper/2305.13047/) (2023)
- [TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation](https://mlsanity.com/paper/2401.14373/) (2024)
- [Lessons learned from the evaluation of Spanish Language Models](https://mlsanity.com/paper/2212.08390/) (2022)
- [Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages](https://mlsanity.com/paper/2112.10553/) (2021)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [Evaluating Transferability of BERT Models on Uralic Languages](https://mlsanity.com/paper/2109.06327/) (2021)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets](https://mlsanity.com/paper/2403.20056/) (2024)
- [Evaluation of contextual embeddings on less-resourced languages](https://mlsanity.com/paper/2107.10614/) (2021)
- [AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese](https://mlsanity.com/paper/2305.13641/) (2023)
- [Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining](https://mlsanity.com/paper/2404.05428/) (2024)
- [LLMs for Extremely Low-Resource Finno-Ugric Languages](https://mlsanity.com/paper/2410.18902/) (2024)
- [Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian](https://mlsanity.com/paper/2501.02631/) (2025)
- [Comparison of Current Approaches to Lemmatization: A Case Study in Estonian](https://mlsanity.com/paper/2404.15003/) (2024)
- [Comparing Approaches to Automatic Summarization in Less-Resourced Languages](https://mlsanity.com/paper/2512.24410/) (2025)

## Abstract (arXiv)

This paper presents EstBERT, a large pretrained transformer-based language-specific BERT model for Estonian. Recent work has evaluated multilingual BERT models on Estonian tasks and found them to outperform the baselines. Still, based on existing studies on other languages, a language-specific BERT model is expected to improve over the multilingual ones. We first describe the EstBERT pretraining process and then present the results of the models based on finetuned EstBERT for multiple NLP tasks, including POS and morphological tagging, named entity recognition and text classification. The evaluation results show that the models based on EstBERT outperform multilingual BERT models on five tasks out of six, providing further evidence towards a view that training language-specific BERT models are still useful, even when multilingual models are available.