# Analyzing COVID-19 Tweets with Transformer-based Language Models

arXiv: https://arxiv.org/abs/2104.10259 · HTML: https://mlsanity.com/paper/2104.10259/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [The Radicalization Risks of GPT-3 and Advanced Neural Language Models](https://mlsanity.com/paper/2009.06807/) (2020)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [Navigating Human Language Models with Synthetic Agents](https://mlsanity.com/paper/2008.04162/) (2020)

## Built on by (most cited first, 5 of 5)

- [CommunityLM: Probing Partisan Worldviews from Language Models](https://mlsanity.com/paper/2209.07065/) (2022)
- [Mixed-effects transformers for hierarchical adaptation](https://mlsanity.com/paper/2205.01749/) (2022)
- [Ethics, Rules of Engagement, and AI: Neural Narrative Mapping Using Large Transformer Language Models](https://mlsanity.com/paper/2202.02647/) (2022)
- [Beyond Accuracy: Community Perspectives on Machine Translation](https://mlsanity.com/paper/2606.09655/) (2026)
- [When Infodemic Meets Epidemic: a Systematic Literature Review](https://mlsanity.com/paper/2210.04612/) (2022)

## Abstract (arXiv)

This paper describes a method for using Transformer-based Language Models (TLMs) to understand public opinion from social media posts. In this approach, we train a set of GPT models on several COVID-19 tweet corpora that reflect populations of users with distinctive views. We then use prompt-based queries to probe these models to reveal insights into the biases and opinions of the users. We demonstrate how this approach can be used to produce results which resemble polling the public on diverse social, political and public health issues. The results on the COVID-19 tweet data show that transformer language models are promising tools that can help us understand public opinions on social media at scale.