# Evaluating the Robustness of Adverse Drug Event Classification Models Using Templates

arXiv: https://arxiv.org/abs/2407.02432 · HTML: https://mlsanity.com/paper/2407.02432/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [COMETA: A Corpus for Medical Entity Linking in the Social Media](https://mlsanity.com/paper/2010.03295/) (2020)
- [Supporting Human-AI Collaboration in Auditing LLMs with LLMs](https://mlsanity.com/paper/2304.09991/) (2023)
- [An Analysis of Negation in Natural Language Understanding Corpora](https://mlsanity.com/paper/2203.08929/) (2022)
- [Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation](https://mlsanity.com/paper/2210.03256/) (2022)
- [Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates](https://mlsanity.com/paper/2105.03048/) (2021)
- [Improving the robustness and accuracy of biomedical language models through adversarial training](https://mlsanity.com/paper/2111.08529/) (2021)
- [NADE: A Benchmark for Robust Adverse Drug Events Extraction in Face of Negations](https://mlsanity.com/paper/2109.10080/) (2021)

## Built on by (most cited first, 1 of 1)

- [DeVisE: Behavioral Testing of Medical Large Language Models](https://mlsanity.com/paper/2506.15339/) (2025)

## Abstract (arXiv)

An adverse drug effect (ADE) is any harmful event resulting from medical drug treatment. Despite their importance, ADEs are often under-reported in official channels. Some research has therefore turned to detecting discussions of ADEs in social media. Impressive results have been achieved in various attempts to detect ADEs. In a high-stakes domain such as medicine, however, an in-depth evaluation of a model's abilities is crucial. We address the issue of thorough performance evaluation in English-language ADE detection with hand-crafted templates for four capabilities: Temporal order, negation, sentiment, and beneficial effect. We find that models with similar performance on held-out test sets have varying results on these capabilities.