# BERTi'c -- The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbian

arXiv: https://arxiv.org/abs/2104.09243 · HTML: https://mlsanity.com/paper/2104.09243/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)

## Built on by (most cited first, 21 of 21)

- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [NollySenti: Leveraging Transfer Learning and Machine Translation for Nigerian Movie Sentiment Classification](https://mlsanity.com/paper/2305.10971/) (2023)
- [Multilingual transformer and BERTopic for short text topic modeling: The case of Serbian](https://mlsanity.com/paper/2402.03067/) (2024)
- [Evaluation of contextual embeddings on less-resourced languages](https://mlsanity.com/paper/2107.10614/) (2021)
- [CoRAL: a Context-aware Croatian Abusive Language Dataset](https://mlsanity.com/paper/2211.06053/) (2022)
- [Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages](https://mlsanity.com/paper/2403.08693/) (2024)
- [Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining](https://mlsanity.com/paper/2404.05428/) (2024)
- [LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma](https://mlsanity.com/paper/2403.00418/) (2024)
- [Data-Augmentation-Based Dialectal Adaptation for LLMs](https://mlsanity.com/paper/2404.08092/) (2024)
- [The ParlaSent-BCS dataset of sentiment-annotated parliamentary debates from Bosnia-Herzegovina, Croatia, and Serbia](https://mlsanity.com/paper/2206.00929/) (2022)
- [EENLP: Cross-lingual Eastern European NLP Index](https://mlsanity.com/paper/2108.02605/) (2021)
- [New Textual Corpora for Serbian Language Modeling](https://mlsanity.com/paper/2405.09250/) (2024)
- [Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices](https://mlsanity.com/paper/2510.24450/) (2025)
- [Lyapunov Spectral Analysis of Speech Embedding Trajectories in Psychosis](https://mlsanity.com/paper/2602.16273/) (2026)
- [A Survey of Resources and Methods for Natural Language Processing of Serbian Language](https://mlsanity.com/paper/2304.05468/) (2023)
- [Croatian Film Review Dataset (Cro-FiReDa): A Sentiment Annotated Dataset of Film Reviews](https://mlsanity.com/paper/2305.08173/) (2023)
- [Novi jezivcki modeli za srpski jezik](https://mlsanity.com/paper/2402.14379/) (2024)
- [Synthetic Dataset Creation and Fine-Tuning of Transformer Models for Question Answering in Serbian](https://mlsanity.com/paper/2404.08617/) (2024)
- [Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language](https://mlsanity.com/paper/2506.09560/) (2025)
- [Characterizing Linguistic Shifts in Croatian News via Diachronic Word Embeddings](https://mlsanity.com/paper/2506.13569/) (2025)
- [What Makes You CLIC: Detection of Croatian Clickbait Headlines](https://mlsanity.com/paper/2507.14314/) (2025)

## Abstract (arXiv)

In this paper we describe a transformer model pre-trained on 8 billion tokens of crawled text from the Croatian, Bosnian, Serbian and Montenegrin web domains. We evaluate the transformer model on the tasks of part-of-speech tagging, named-entity-recognition, geo-location prediction and commonsense causal reasoning, showing improvements on all tasks over state-of-the-art models. For commonsense reasoning evaluation, we introduce COPA-HR -- a translation of the Choice of Plausible Alternatives (COPA) dataset into Croatian. The BERTi'c model is made available for free usage and further task-specific fine-tuning through HuggingFace.