# Exploring a Unified Sequence-To-Sequence Transformer for Medical Product Safety Monitoring in Social Media

arXiv: https://arxiv.org/abs/2109.05815 · HTML: https://mlsanity.com/paper/2109.05815/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [An Effective Transition-based Model for Discontinuous NER](https://mlsanity.com/paper/2004.13454/) (2020)
- [Improving Adverse Drug Event Extraction with SpanBERT on Different Text Typologies](https://mlsanity.com/paper/2105.08882/) (2021)

## Built on by (most cited first, 9 of 9)

- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Extensive Evaluation of Transformer-based Architectures for Adverse Drug Events Extraction](https://mlsanity.com/paper/2306.05276/) (2023)
- [Cross-lingual Approaches for the Detection of Adverse Drug Reactions in German from a Patient's Perspective](https://mlsanity.com/paper/2208.02031/) (2022)
- [MultiADE: A Multi-domain Benchmark for Adverse Drug Event Extraction](https://mlsanity.com/paper/2405.18015/) (2024)
- [GPT-3 Powered Information Extraction for Building Robust Knowledge Bases](https://mlsanity.com/paper/2408.04641/) (2024)
- [KESDT: knowledge enhanced shallow and deep Transformer for detecting adverse drug reactions](https://mlsanity.com/paper/2308.09329/) (2023)
- [A Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages](https://mlsanity.com/paper/2403.18336/) (2024)
- [ADEQA: A Question Answer based approach for joint ADE-Suspect Extraction using Sequence-To-Sequence Transformers](https://mlsanity.com/paper/2412.15510/) (2024)

## Abstract (arXiv)

Adverse Events (AE) are harmful events resulting from the use of medical products. Although social media may be crucial for early AE detection, the sheer scale of this data makes it logistically intractable to analyze using human agents, with NLP representing the only low-cost and scalable alternative. In this paper, we frame AE Detection and Extraction as a sequence-to-sequence problem using the T5 model architecture and achieve strong performance improvements over competitive baselines on several English benchmarks (F1 = 0.71, 12.7% relative improvement for AE Detection; Strict F1 = 0.713, 12.4% relative improvement for AE Extraction). Motivated by the strong commonalities between AE-related tasks, the class imbalance in AE benchmarks and the linguistic and structural variety typical of social media posts, we propose a new strategy for multi-task training that accounts, at the same time, for task and dataset characteristics. Our multi-task approach increases model robustness, leading to further performance gains. Finally, our framework shows some language transfer capabilities, obtaining higher performance than Multilingual BERT in zero-shot learning on French data.