# Stress Test Evaluation of Biomedical Word Embeddings

arXiv: https://arxiv.org/abs/2107.11652 · HTML: https://mlsanity.com/paper/2107.11652/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 24 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems](https://mlsanity.com/paper/1907.10456/) (2019)
- [Probing Biomedical Embeddings from Language Models](https://mlsanity.com/paper/1904.02181/) (2019)
- [Generalizability vs. Robustness: Adversarial Examples for Medical Imaging](https://mlsanity.com/paper/1804.00504/) (2018)
- [Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks](https://mlsanity.com/paper/2002.06261/) (2020)
- [Identify Susceptible Locations in Medical Records via Adversarial Attacks on Deep Predictive Models](https://mlsanity.com/paper/1802.04822/) (2018)
- [Improving Chemical Named Entity Recognition in Patents with Contextualized Word Embeddings](https://mlsanity.com/paper/1907.02679/) (2019)

## Built on by (most cited first, 4 of 4)

- [Evaluation Benchmarks for Spanish Sentence Representations](https://mlsanity.com/paper/2204.07571/) (2022)
- [Robustness Stress Testing in Medical Image Classification](https://mlsanity.com/paper/2308.06889/) (2023)
- [User Perception of Attention Visualizations: Effects on Interpretability Across Evidence-Based Medical Documents](https://mlsanity.com/paper/2508.10004/) (2025)
- [Stress Testing BERT Anaphora Resolution Models for Reaction Extraction in Chemical Patents](https://mlsanity.com/paper/2306.13379/) (2023)

## Abstract (arXiv)

The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research on quantifying their behavior under severe "stress" scenarios. In this work, we systematically evaluate three language models with adversarial examples -- automatically constructed tests that allow us to examine how robust the models are. We propose two types of stress scenarios focused on the biomedical named entity recognition (NER) task, one inspired by spelling errors and another based on the use of synonyms for medical terms. Our experiments with three benchmarks show that the performance of the original models decreases considerably, in addition to revealing their weaknesses and strengths. Finally, we show that adversarial training causes the models to improve their robustness and even to exceed the original performance in some cases.