# The birth of Romanian BERT

arXiv: https://arxiv.org/abs/2009.08712 · HTML: https://mlsanity.com/paper/2009.08712/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2020.

## Built on by (most cited first, 30 of 44)

- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost](https://mlsanity.com/paper/2211.06993/) (2022)
- [Czert -- Czech BERT-like Model for Language Representation](https://mlsanity.com/paper/2103.13031/) (2021)
- [Distilling the Knowledge of Romanian BERTs Using Multiple Teachers](https://mlsanity.com/paper/2112.12650/) (2021)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articles](https://mlsanity.com/paper/2105.06456/) (2021)
- [Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages](https://mlsanity.com/paper/2303.17683/) (2023)
- [Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa -- A Large Romanian Sentiment Data Set](https://mlsanity.com/paper/2101.04197/) (2021)
- [A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus](https://mlsanity.com/paper/2405.11877/) (2024)
- [Are the Multilingual Models Better? Improving Czech Sentiment with Transformers](https://mlsanity.com/paper/2108.10640/) (2021)
- [RoQLlama: A Lightweight Romanian Adapted Language Model](https://mlsanity.com/paper/2410.04269/) (2024)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model](https://mlsanity.com/paper/2109.06605/) (2021)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [HistNERo: Historical Named Entity Recognition for the Romanian Language](https://mlsanity.com/paper/2405.00155/) (2024)
- [Romanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibition](https://mlsanity.com/paper/2304.11350/) (2023)
- [Adversarial Capsule Networks for Romanian Satire Detection and Sentiment Analysis](https://mlsanity.com/paper/2306.07845/) (2023)
- [A Novel Contrastive Learning Method for Clickbait Detection on RoCliCo: A Romanian Clickbait Corpus of News Articles](https://mlsanity.com/paper/2310.06540/) (2023)
- [The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages](https://mlsanity.com/paper/2310.14557/) (2023)
- [RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian](https://mlsanity.com/paper/2402.13222/) (2024)
- [Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature](https://mlsanity.com/paper/2601.07533/) (2026)
- [BERT-based Authorship Attribution on the Romanian Dataset called ROST](https://mlsanity.com/paper/2301.12500/) (2023)
- [Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptation](https://mlsanity.com/paper/2306.10419/) (2023)
- [Cascaded Cross-Modal Transformer for Audio-Textual Classification](https://mlsanity.com/paper/2401.07575/) (2024)
- [Romanian Speech Recognition Experiments from the ROBIN Project](https://mlsanity.com/paper/2111.12028/) (2021)
- [Go-tuning: Improving Zero-shot Learning Abilities of Smaller Language Models](https://mlsanity.com/paper/2212.10461/) (2022)
- [RALS: Resources and Baselines for Romanian Automatic Lexical Simplification](https://mlsanity.com/paper/2607.20078/) (2026)
- [Dual-State Capsule Networks for Text Classification](https://mlsanity.com/paper/2109.04762/) (2021)

## Abstract (arXiv)

Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.