# COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter

arXiv: https://arxiv.org/abs/2005.07503 · HTML: https://mlsanity.com/paper/2005.07503/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2020.

## Built on by (most cited first, 30 of 96)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation](https://mlsanity.com/paper/2310.15515/) (2023)
- [Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines](https://mlsanity.com/paper/2104.08790/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [The Surprising Performance of Simple Baselines for Misinformation Detection](https://mlsanity.com/paper/2104.06952/) (2021)
- [HealthPrompt: A Zero-shot Learning Paradigm for Clinical Natural Language Processing](https://mlsanity.com/paper/2203.05061/) (2022)
- [Infusing Knowledge from Wikipedia to Enhance Stance Detection](https://mlsanity.com/paper/2204.03839/) (2022)
- [Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels](https://mlsanity.com/paper/2311.05265/) (2023)
- [Evaluating Deep Learning Approaches for Covid19 Fake News Detection](https://mlsanity.com/paper/2101.04012/) (2021)
- [CAVES: A Dataset to facilitate Explainable Classification and Summarization of Concerns towards COVID Vaccines](https://mlsanity.com/paper/2204.13746/) (2022)
- [Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection](https://mlsanity.com/paper/2101.05509/) (2021)
- [Contrastive Domain Adaptation for Early Misinformation Detection: A Case Study on COVID-19](https://mlsanity.com/paper/2208.09578/) (2022)
- [Knowledge Graph informed Fake News Classification via Heterogeneous Representation Ensembles](https://mlsanity.com/paper/2110.10457/) (2021)
- [Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments](https://mlsanity.com/paper/2212.09683/) (2022)
- [VaxxHesitancy: A Dataset for Studying Hesitancy towards COVID-19 Vaccination on Twitter](https://mlsanity.com/paper/2301.06660/) (2023)
- [Examining Bias in Opinion Summarisation Through the Perspective of Opinion Diversity](https://mlsanity.com/paper/2306.04424/) (2023)
- [A Comparative Study of Transformer-Based Language Models on Extractive Question Answering](https://mlsanity.com/paper/2110.03142/) (2021)
- [METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets](https://mlsanity.com/paper/2209.13773/) (2022)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection](https://mlsanity.com/paper/2111.07819/) (2021)
- [QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search](https://mlsanity.com/paper/2306.06707/) (2023)
- [ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking](https://mlsanity.com/paper/2510.13842/) (2025)
- [Categorising Fine-to-Coarse Grained Misinformation: An Empirical Study of COVID-19 Infodemic](https://mlsanity.com/paper/2106.11702/) (2021)
- [Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning](https://mlsanity.com/paper/2305.08473/) (2023)
- [Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets](https://mlsanity.com/paper/2309.06503/) (2023)
- [VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding](https://mlsanity.com/paper/2403.15882/) (2024)

## Abstract (arXiv)

In this work, we release COVID-Twitter-BERT (CT-BERT), a transformer-based model, pretrained on a large corpus of Twitter messages on the topic of COVID-19. Our model shows a 10-30% marginal improvement compared to its base model, BERT-Large, on five different classification datasets. The largest improvements are on the target domain. Pretrained transformer models, such as CT-BERT, are trained on a specific target domain and can be used for a wide variety of natural language processing tasks, including classification, question-answering and chatbots. CT-BERT is optimised to be used on COVID-19 content, in particular social media posts from Twitter.