# RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media

arXiv: https://arxiv.org/abs/2210.06331 · HTML: https://mlsanity.com/paper/2210.06331/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature](https://mlsanity.com/paper/1806.04185/) (2018)
- [SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions](https://mlsanity.com/paper/1806.05258/) (2018)
- [Inferring Which Medical Treatments Work from Reports of Clinical Trials](https://mlsanity.com/paper/1904.01606/) (2019)
- [UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification](https://mlsanity.com/paper/1809.01479/) (2018)
- [COMETA: A Corpus for Medical Entity Linking in the Social Media](https://mlsanity.com/paper/2010.03295/) (2020)
- [Scientific Claim Verification with VERT5ERINI](https://mlsanity.com/paper/2010.11930/) (2020)
- [Evidence Inference 2.0: More Data, Better Models](https://mlsanity.com/paper/2005.04177/) (2020)
- [RELIC: Retrieving Evidence for Literary Claims](https://mlsanity.com/paper/2203.10053/) (2022)
- [MIMICause: Representation and automatic extraction of causal relation types from clinical notes](https://mlsanity.com/paper/2110.07090/) (2021)
- [Determining Question-Answer Plausibility in Crowdsourced Datasets Using Multi-Task Learning](https://mlsanity.com/paper/2011.04883/) (2020)
- [Microtask crowdsourcing for disease mention annotation in PubMed abstracts](https://mlsanity.com/paper/1408.1928/) (2014)

## Built on by (most cited first, 13 of 13)

- [HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checking](https://mlsanity.com/paper/2309.08503/) (2023)
- [Comparing Knowledge Sources for Open-Domain Scientific Claim Verification](https://mlsanity.com/paper/2402.02844/) (2024)
- [Combating Health Misinformation in Social Media: Characterization, Detection, Intervention, and Open Issues](https://mlsanity.com/paper/2211.05289/) (2022)
- [AUTALIC: A Dataset for Anti-AUTistic Ableist Language In Context](https://mlsanity.com/paper/2410.16520/) (2024)
- [Beyond True or False: Retrieval-Augmented Hierarchical Analysis of Nuanced Claims](https://mlsanity.com/paper/2506.10728/) (2025)
- [Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence](https://mlsanity.com/paper/2601.11886/) (2026)
- [Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine](https://mlsanity.com/paper/2506.20876/) (2025)
- [This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA](https://mlsanity.com/paper/2604.05051/) (2026)
- [CAISA at SemEval-2023 Task 8: Counterfactual Data Augmentation for Mitigating Class Imbalance in Causal Claim Identification](https://mlsanity.com/paper/2306.00346/) (2023)
- [MasonNLP+ at SemEval-2023 Task 8: Extracting Medical Questions, Experiences and Claims from Social Media using Knowledge-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2304.13875/) (2023)
- [Identifying and Aligning Medical Claims Made on Social Media with Medical Evidence](https://mlsanity.com/paper/2405.11219/) (2024)
- [Health Misinformation in Social Networks: A Survey of IT Approaches](https://mlsanity.com/paper/2410.18670/) (2024)
- [Towards Expectation Detection in Language: A Case Study on Treatment Expectations in Reddit](https://mlsanity.com/paper/2602.15504/) (2026)

## Abstract (arXiv)

We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly annotated social media posts from Reddit spanning 24 health conditions. Annotations include demarcations of spans corresponding to medical claims, personal experiences, and questions. We collect additional granular annotations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence relevant to a given claim made on social media. We propose a new method to automatically derive (noisy) supervision for this task which we use to train a dense retrieval model; this outperforms baseline models. Manual evaluation of retrieval results performed by medical doctors indicate that while our system performance is promising, there is considerable room for improvement. Collected annotations (and scripts to assemble the dataset), are available at https://github.com/sominw/redhot.