mlsanityData to 7 Oct 2026

FinEst BERT and CroSloEngual BERT: less is more in multilingual models

arXiv 2006.07890 ↗First posted 14 Jun 2020cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. The research has been mostly focused on English language, though. While massively multilingual models exist, studies have shown that monolingual models produce much better results. We train two trilingual BERT-like models, one for Finnish, Estonian, and English, the other for Croatian, Slovenian, and English. We evaluate their performance on several downstream tasks, NER, POS-tagging, and dependency parsing, using the multilingual BERT and XLM-R as baselines. The newly created FinEst BERT and CroSloEngual BERT improve the results on all tasks in most monolingual and cross-lingual situations

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 4; 2024: 4; 2025: 1; 2026: 1.

20 papers.

Sort:

  1. 2024Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddingscited by 20
  2. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  3. 2022Sequence to sequence pretraining for a less-resourced Slovenian languagecited by 9
  4. 2023Developing and Evaluating Tiny to Medium-Sized Turkish BERT Modelscited by 8
  5. 2021Training dataset and dictionary sizes matter in BERT models: the case of Baltic languagescited by 7
  6. 2022Slovene SuperGLUE Benchmark: Translation and Evaluationcited by 7
  7. 2024Generative Model for Less-Resourced Language with 1 billion parameterscited by 5
  8. 2021Evaluation of contextual embeddings on less-resourced languagescited by 4
  9. 2022CoRAL: a Context-aware Croatian Abusive Language Datasetcited by 4
  10. 2024Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretrainingcited by 4
  11. 2024Code-mixed Sentiment and Hate-speech Predictioncited by 4
  12. 2021JABER and SABER: Junior and Senior Arabic BERtcited by 3
8 more
  1. 2022Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understandingcited by 2
  2. 2022The ParlaSent-BCS dataset of sentiment-annotated parliamentary debates from Bosnia-Herzegovina, Croatia, and Serbiacited by 2
  3. 2021EENLP: Cross-lingual Eastern European NLP Indexcited by 1
  4. 2026Building a Strong Instruction Language Model for a Less-Resourced Languagecited by 1
  5. 2023A Survey of Resources and Methods for Natural Language Processing of Serbian Languagecited by 0
  6. 2023Croatian Film Review Dataset (Cro-FiReDa): A Sentiment Annotated Dataset of Film Reviewscited by 0
  7. 2023CroSentiNews 2.0: A Sentence-Level News Sentiment Corpuscited by 0
  8. 2025Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Languagecited by 0