# Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model

arXiv: https://arxiv.org/abs/2204.04521 · HTML: https://mlsanity.com/paper/2204.04521/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 9 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019)
- [A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models](https://mlsanity.com/paper/2010.15036/) (2020)
- [Latent Suicide Risk Detection on Microblog via Suicide-Oriented Word Embeddings and Layered Attention](https://mlsanity.com/paper/1910.12038/) (2019)
- [ktrain: A Low-Code Library for Augmented Machine Learning](https://mlsanity.com/paper/2004.10703/) (2020)
- [Did You Really Just Have a Heart Attack? Towards Robust Detection of Personal Health Mentions in Social Media](https://mlsanity.com/paper/1802.09130/) (2018)
- [BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition](https://mlsanity.com/paper/2009.09223/) (2020)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Crowdbreaks: Tracking Health Trends using Public Social Media Data and Crowdsourcing](https://mlsanity.com/paper/1805.05491/) (2018)
- [Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU](https://mlsanity.com/paper/2106.09589/) (2021)

## Built on by (most cited first, 7 of 7)

- [Rethinking Large Language Models in Mental Health Applications](https://mlsanity.com/paper/2311.11267/) (2023)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlook](https://mlsanity.com/paper/2410.08793/) (2024)
- [A Unique Training Strategy to Enhance Language Models Capabilities for Health Mention Detection from Social Media Content](https://mlsanity.com/paper/2310.19057/) (2023)
- [LocalTweets to LocalHealth: A Mental Health Surveillance Framework Based on Twitter Data](https://mlsanity.com/paper/2402.13452/) (2024)
- [Improving Disease Detection from Social Media Text via Self-Augmentation and Contrastive Learning](https://mlsanity.com/paper/2405.01597/) (2024)
- [Health Disparities through Generative AI Models: A Comparison Study Using A Domain Specific large language model](https://mlsanity.com/paper/2310.18355/) (2023)

## Abstract (arXiv)

A user-generated text on social media enables health workers to keep track of information, identify possible outbreaks, forecast disease trends, monitor emergency cases, and ascertain disease awareness and response to official health correspondence. This exchange of health information on social media has been regarded as an attempt to enhance public health surveillance (PHS). Despite its potential, the technology is still in its early stages and is not ready for widespread application. Advancements in pretrained language models (PLMs) have facilitated the development of several domain-specific PLMs and a variety of downstream applications. However, there are no PLMs for social media tasks involving PHS. We present and release PHS-BERT, a transformer-based PLM, to identify tasks related to public health surveillance on social media. We compared and benchmarked the performance of PHS-BERT on 25 datasets from different social medial platforms related to 7 different PHS tasks. Compared with existing PLMs that are mainly evaluated on limited tasks, PHS-BERT achieved state-of-the-art performance on all 25 tested datasets, showing that our PLM is robust and generalizable in the common PHS tasks. By making PHS-BERT available, we aim to facilitate the community to reduce the computational cost and introduce new baselines for future works across various PHS-related tasks.