# RoBERTweet: A BERT Language Model for Romanian Tweets

arXiv: https://arxiv.org/abs/2306.06598 · HTML: https://mlsanity.com/paper/2306.06598/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter](https://mlsanity.com/paper/2209.07562/) (2022)
- [The birth of Romanian BERT](https://mlsanity.com/paper/2009.08712/) (2020)
- [IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization](https://mlsanity.com/paper/2109.04607/) (2021)
- [RoBERTuito: a pre-trained language model for social media text in Spanish](https://mlsanity.com/paper/2111.09453/) (2021)
- [Distilling the Knowledge of Romanian BERTs Using Multiple Teachers](https://mlsanity.com/paper/2112.12650/) (2021)
- [Language Variety Identification with True Labels](https://mlsanity.com/paper/2303.01490/) (2023)
- [BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets](https://mlsanity.com/paper/2109.10234/) (2021)

## Abstract (arXiv)

Developing natural language processing (NLP) systems for social media analysis remains an important topic in artificial intelligence research. This article introduces RoBERTweet, the first Transformer architecture trained on Romanian tweets. Our RoBERTweet comes in two versions, following the base and large architectures of BERT. The corpus used for pre-training the models represents a novelty for the Romanian NLP community and consists of all tweets collected from 2008 to 2022. Experiments show that RoBERTweet models outperform the previous general-domain Romanian and multilingual language models on three NLP tasks with tweet inputs: emotion detection, sexist language identification, and named entity recognition. We make our models and the newly created corpus of Romanian tweets freely available.