# TweetBERT: A Pretrained Language Representation Model for Twitter Text Analysis

arXiv: https://arxiv.org/abs/2010.11091 · HTML: https://mlsanity.com/paper/2010.11091/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data](https://mlsanity.com/paper/2403.19031/) (2024)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [A Survey on Automated Sarcasm Detection on Twitter](https://mlsanity.com/paper/2202.02516/) (2022)
- [Domain-oriented Language Pre-training with Adaptive Hybrid Masking and Optimal Transport Alignment](https://mlsanity.com/paper/2112.03024/) (2021)
- [NTULM: Enriching Social Media Text Representations with Non-Textual Units](https://mlsanity.com/paper/2210.16586/) (2022)
- [Personality Detection and Analysis using Twitter Data](https://mlsanity.com/paper/2309.05497/) (2023)
- [Meta learning with language models: Challenges and opportunities in the classification of imbalanced text](https://mlsanity.com/paper/2310.15019/) (2023)
- [Automating Transfer Credit Assessment in Student Mobility -- A Natural Language Processing-based Approach](https://mlsanity.com/paper/2104.01955/) (2021)
- [Extraction of Medication Names from Twitter Using Augmentation and an Ensemble of Language Models](https://mlsanity.com/paper/2111.06664/) (2021)
- [DeepTrust: A Reliable Financial Knowledge Retrieval Framework For Explaining Extreme Pricing Anomalies](https://mlsanity.com/paper/2203.08144/) (2022)
- [Limits to Predicting Online Speech Using Large Language Models](https://mlsanity.com/paper/2407.12850/) (2024)
- [A Mechanism and Optimization Study on the Impact of Information Density on User-Generated Content Named Entity Recognition](https://mlsanity.com/paper/2604.18944/) (2026)

## Abstract (arXiv)

Twitter is a well-known microblogging social site where users express their views and opinions in real-time. As a result, tweets tend to contain valuable information. With the advancements of deep learning in the domain of natural language processing, extracting meaningful information from tweets has become a growing interest among natural language researchers. Applying existing language representation models to extract information from Twitter does not often produce good results. Moreover, there is no existing language representation models for text analysis specific to the social media domain. Hence, in this article, we introduce two TweetBERT models, which are domain specific language presentation models, pre-trained on millions of tweets. We show that the TweetBERT models significantly outperform the traditional BERT models in Twitter text mining tasks by more than 7% on each Twitter dataset. We also provide an extensive analysis by evaluating seven BERT models on 31 different datasets. Our results validate our hypothesis that continuously training language models on twitter corpus help performance with Twitter.