mlsanityData to 7 Oct 2026

The birth of Romanian BERT

arXiv 2009.08712 ↗First posted 18 Sep 2020cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 4; 2023: 11; 2024: 7; 2025: 6; 2026: 5.

44 papers.

Sort:

  1. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  4. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  5. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  6. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teacherscited by 13
  7. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  8. 2021SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articlescited by 12
  9. 2023Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languagescited by 9
  10. 2021Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa -- A Large Romanian Sentiment Data Setcited by 8
  11. 2024A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpuscited by 8
  12. 2021Are the Multilingual Models Better? Improving Czech Sentiment with Transformerscited by 7
32 more
  1. 2024RoQLlama: A Lightweight Romanian Adapted Language Modelcited by 7
  2. 2021PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptorscited by 6
  3. 2021MDAPT: Multilingual Domain Adaptive Pretraining in a Single Modelcited by 6
  4. 2023Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languagescited by 6
  5. 2024HistNERo: Historical Named Entity Recognition for the Romanian Languagecited by 5
  6. 2023Romanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibitioncited by 4
  7. 2023Adversarial Capsule Networks for Romanian Satire Detection and Sentiment Analysiscited by 3
  8. 2023A Novel Contrastive Learning Method for Clickbait Detection on RoCliCo: A Romanian Clickbait Corpus of News Articlescited by 3
  9. 2023The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languagescited by 3
  10. 2024RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romaniancited by 3
  11. 2026Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literaturecited by 3
  12. 2023BERT-based Authorship Attribution on the Romanian Dataset called ROSTcited by 2
  13. 2023Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptationcited by 2
  14. 2024Cascaded Cross-Modal Transformer for Audio-Textual Classificationcited by 2
  15. 2021Romanian Speech Recognition Experiments from the ROBIN Projectcited by 1
  16. 2022Go-tuning: Improving Zero-shot Learning Abilities of Smaller Language Modelscited by 1
  17. 2026RALS: Resources and Baselines for Romanian Automatic Lexical Simplificationcited by 1
  18. 2021Dual-State Capsule Networks for Text Classificationcited by 0
  19. 2022Punctuation restoration in Swedish through fine-tuned KB-BERTcited by 0
  20. 2022An Open-Domain QA System for e-Governancecited by 0
  21. 2023RoBERTweet: A BERT Language Model for Romanian Tweetscited by 0
  22. 2024Human and Automatic Interpretation of Romanian Noun Compoundscited by 0
  23. 2024PoPreRo: A New Dataset for Popularity Prediction of Romanian Reddit Postscited by 0
  24. 2025UniBERT: Adversarial Training for Language-Universal Representationscited by 0
  25. 2025SaRoHead: Detecting Satire in a Multi-Domain Romanian News Headline Datasetcited by 0
  26. 2025MuSaRoNews: A Multidomain, Multimodal Satire Dataset from Romanian News Articlescited by 0
  27. 2025A Culturally-Rich Romanian NLP Dataset from "Who Wants to Be a Millionaire?" Videoscited by 0
  28. 2025SeLeRoSa: Sentence-Level Romanian Satire Detection Datasetcited by 0
  29. 2025Improving Romanian LLM Pretraining Data using Diversity and Quality Filteringcited by 0
  30. 2026PsihoRo: Depression and Anxiety Romanian Text Corpuscited by 0
  31. 2026Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romaniancited by 0
  32. 2026PolERo: Studying Political Evasion in Romaniancited by 0