# Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning

arXiv: https://arxiv.org/abs/2012.15699 · HTML: https://mlsanity.com/paper/2012.15699/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 30 of 39)

- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization](https://mlsanity.com/paper/2212.09254/) (2022)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Textual Manifold-based Defense Against Natural Language Adversarial Examples](https://mlsanity.com/paper/2211.02878/) (2022)
- [Robust Lottery Tickets for Pre-trained Language Models](https://mlsanity.com/paper/2211.03013/) (2022)
- [On Robust Prefix-Tuning for Text Classification](https://mlsanity.com/paper/2203.10378/) (2022)
- [Less is More: Understanding Word-level Textual Adversarial Attack via n-gram Frequency Descend](https://mlsanity.com/paper/2302.02568/) (2023)
- [Model-tuning Via Prompts Makes NLP Models Adversarially Robust](https://mlsanity.com/paper/2303.07320/) (2023)
- [EPiDA: An Easy Plug-in Data Augmentation Framework for High Performance Text Classification](https://mlsanity.com/paper/2204.11205/) (2022)
- [PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learners](https://mlsanity.com/paper/2205.09229/) (2022)
- [DSRM: Boost Textual Adversarial Training with Distribution Shift Risk Minimization](https://mlsanity.com/paper/2306.15164/) (2023)
- [ROSE: Robust Selective Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2210.09658/) (2022)
- [Cyberbullying Classifiers are Sensitive to Model-Agnostic Perturbations](https://mlsanity.com/paper/2201.06384/) (2022)
- [A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models](https://mlsanity.com/paper/2402.11469/) (2024)
- [Improving robustness of language models from a geometry-aware perspective](https://mlsanity.com/paper/2204.13309/) (2022)
- [Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learning](https://mlsanity.com/paper/2405.15984/) (2024)
- [Robust Text Classification: Analyzing Prototype-Based Networks](https://mlsanity.com/paper/2311.06647/) (2023)
- [Adapters Mixup: Mixing Parameter-Efficient Adapters to Enhance the Adversarial Robustness of Fine-tuned Pre-trained Text Classifiers](https://mlsanity.com/paper/2401.10111/) (2024)
- [ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation](https://mlsanity.com/paper/2210.12396/) (2022)
- [Enhance Robustness of Language Models Against Variation Attack through Graph Integration](https://mlsanity.com/paper/2404.12014/) (2024)
- [MaskPure: Improving Defense Against Text Adversaries with Stochastic Purification](https://mlsanity.com/paper/2406.13066/) (2024)
- [Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks](https://mlsanity.com/paper/2501.02654/) (2025)
- [BootAug: Boosting Text Augmentation via Hybrid Instance Filtering Framework](https://mlsanity.com/paper/2210.02941/) (2022)
- [RoChBert: Towards Robust BERT Fine-tuning for Chinese](https://mlsanity.com/paper/2210.15944/) (2022)
- [Fooling the Textual Fooler via Randomizing Latent Representations](https://mlsanity.com/paper/2310.01452/) (2023)
- [Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentation](https://mlsanity.com/paper/2403.15512/) (2024)

## Abstract (arXiv)

Pretrained language models (PLMs) perform poorly under adversarial attacks. To improve the adversarial robustness, adversarial data augmentation (ADA) has been widely adopted to cover more search space of adversarial attacks by adding textual adversarial examples during training. However, the number of adversarial examples for text augmentation is still extremely insufficient due to the exponentially large attack search space. In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA). Specifically, AMDA linearly interpolates the representations of pairs of training samples to form new virtual samples, which are more abundant and diverse than the discrete text adversarial examples in conventional ADA. Moreover, to fairly evaluate the robustness of different models, we adopt a challenging evaluation setup, which generates a new set of adversarial examples targeting each model. In text classification experiments of BERT and RoBERTa, AMDA achieves significant robustness gains under two strong adversarial attacks and alleviates the performance degradation of ADA on the clean data. Our code is available at: https://github.com/thunlp/MixADA .