# Improving the robustness and accuracy of biomedical language models through adversarial training

arXiv: https://arxiv.org/abs/2111.08529 · HTML: https://mlsanity.com/paper/2111.08529/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 62 references that are in the map.
- First posted on arXiv 16 Nov 2021.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [Contextualized Perturbation for Textual Adversarial Attack](https://mlsanity.com/paper/2009.07502/) (2020)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [Construction of the Literature Graph in Semantic Scholar](https://mlsanity.com/paper/1805.02262/) (2018)
- [Deep Text Classification Can be Fooled](https://mlsanity.com/paper/1704.08006/) (2017)
- [Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples](https://mlsanity.com/paper/1803.01128/) (2018)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [OpenAttack: An Open-source Textual Adversarial Attack Toolkit](https://mlsanity.com/paper/2009.09191/) (2020)
- [Interpretable Adversarial Perturbation in Input Embedding Space for Text](https://mlsanity.com/paper/1805.02917/) (2018)
- [Generating Fluent Adversarial Examples for Natural Languages](https://mlsanity.com/paper/2007.06174/) (2020)
- [Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge](https://mlsanity.com/paper/1808.08609/) (2018)

## Built on by (most cited first, 7 of 7)

- [Trustworthy Medical Question Answering: An Evaluation-Centric Survey](https://mlsanity.com/paper/2506.03659/) (2025)
- [Model-agnostic explainable artificial intelligence for object detection in image data](https://mlsanity.com/paper/2303.17249/) (2023)
- [Evaluating the Robustness of Adverse Drug Event Classification Models Using Templates](https://mlsanity.com/paper/2407.02432/) (2024)
- [Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration](https://mlsanity.com/paper/2509.21530/) (2025)
- [Research on Security Enhancement Methods for Adversarial Robust Large Language Model Intelligent Agents for Medical Decision-Making Tasks](https://mlsanity.com/paper/2605.08257/) (2026)
- [Robust Biomedical Publication Type and Study Design Classification with Knowledge-Guided Perturbations](https://mlsanity.com/paper/2605.11502/) (2026)
- [When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations](https://mlsanity.com/paper/2606.07237/) (2026)

## Abstract (arXiv)

Deep transformer neural network models have improved the predictive accuracy of intelligent text processing systems in the biomedical domain. They have obtained state-of-the-art performance scores on a wide variety of biomedical and clinical Natural Language Processing (NLP) benchmarks. However, the robustness and reliability of these models has been less explored so far. Neural NLP models can be easily fooled by adversarial samples, i.e. minor changes to input that preserve the meaning and understandability of the text but force the NLP system to make erroneous decisions. This raises serious concerns about the security and trust-worthiness of biomedical NLP systems, especially when they are intended to be deployed in real-world use cases. We investigated the robustness of several transformer neural language models, i.e. BioBERT, SciBERT, BioMed-RoBERTa, and Bio-ClinicalBERT, on a wide range of biomedical and clinical text processing tasks. We implemented various adversarial attack methods to test the NLP systems in different attack scenarios. Experimental results showed that the biomedical NLP models are sensitive to adversarial samples; their performance dropped in average by 21 and 18.9 absolute percent on character-level and word-level adversarial noise, respectively. Conducting extensive adversarial training experiments, we fine-tuned the NLP models on a mixture of clean samples and adversarial inputs. Results showed that adversarial training is an effective defense mechanism against adversarial noise; the models robustness improved in average by 11.3 absolute percent. In addition, the models performance on clean data increased in average by 2.4 absolute present, demonstrating that adversarial training can boost generalization abilities of biomedical NLP systems.