# FinEst BERT and CroSloEngual BERT: less is more in multilingual models

arXiv: https://arxiv.org/abs/2006.07890 · HTML: https://mlsanity.com/paper/2006.07890/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings](https://mlsanity.com/paper/2402.17016/) (2024)
- [Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition](https://mlsanity.com/paper/2107.11610/) (2021)
- [Sequence to sequence pretraining for a less-resourced Slovenian language](https://mlsanity.com/paper/2207.13988/) (2022)
- [Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models](https://mlsanity.com/paper/2307.14134/) (2023)
- [Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages](https://mlsanity.com/paper/2112.10553/) (2021)
- [Slovene SuperGLUE Benchmark: Translation and Evaluation](https://mlsanity.com/paper/2202.04994/) (2022)
- [Generative Model for Less-Resourced Language with 1 billion parameters](https://mlsanity.com/paper/2410.06898/) (2024)
- [Evaluation of contextual embeddings on less-resourced languages](https://mlsanity.com/paper/2107.10614/) (2021)
- [CoRAL: a Context-aware Croatian Abusive Language Dataset](https://mlsanity.com/paper/2211.06053/) (2022)
- [Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining](https://mlsanity.com/paper/2404.05428/) (2024)
- [Code-mixed Sentiment and Hate-speech Prediction](https://mlsanity.com/paper/2405.12929/) (2024)
- [JABER and SABER: Junior and Senior Arabic BERt](https://mlsanity.com/paper/2112.04329/) (2021)
- [Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding](https://mlsanity.com/paper/2205.10687/) (2022)
- [The ParlaSent-BCS dataset of sentiment-annotated parliamentary debates from Bosnia-Herzegovina, Croatia, and Serbia](https://mlsanity.com/paper/2206.00929/) (2022)
- [EENLP: Cross-lingual Eastern European NLP Index](https://mlsanity.com/paper/2108.02605/) (2021)
- [Building a Strong Instruction Language Model for a Less-Resourced Language](https://mlsanity.com/paper/2603.01691/) (2026)
- [A Survey of Resources and Methods for Natural Language Processing of Serbian Language](https://mlsanity.com/paper/2304.05468/) (2023)
- [Croatian Film Review Dataset (Cro-FiReDa): A Sentiment Annotated Dataset of Film Reviews](https://mlsanity.com/paper/2305.08173/) (2023)
- [CroSentiNews 2.0: A Sentence-Level News Sentiment Corpus](https://mlsanity.com/paper/2305.08187/) (2023)
- [Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language](https://mlsanity.com/paper/2506.09560/) (2025)

## Abstract (arXiv)

Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. The research has been mostly focused on English language, though. While massively multilingual models exist, studies have shown that monolingual models produce much better results. We train two trilingual BERT-like models, one for Finnish, Estonian, and English, the other for Croatian, Slovenian, and English. We evaluate their performance on several downstream tasks, NER, POS-tagging, and dependency parsing, using the multilingual BERT and XLM-R as baselines. The newly created FinEst BERT and CroSloEngual BERT improve the results on all tasks in most monolingual and cross-lingual situations