# SemRoDe: Macro Adversarial Training to Learn Representations That are Robust to Word-Level Attacks

arXiv: https://arxiv.org/abs/2403.18423 · HTML: https://mlsanity.com/paper/2403.18423/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 65 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018)
- [Adversarial Logit Pairing](https://mlsanity.com/paper/1803.06373/) (2018)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [MMA Training: Direct Input Space Margin Maximization through Adversarial Training](https://mlsanity.com/paper/1812.02637/) (2018)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems](https://mlsanity.com/paper/1903.11508/) (2019)
- [InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective](https://mlsanity.com/paper/2010.02329/) (2020)
- [Towards Improving Adversarial Training of NLP Models](https://mlsanity.com/paper/2109.00544/) (2021)
- [SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/2005.14424/) (2020)
- [Towards Robustness Against Natural Language Word Substitutions](https://mlsanity.com/paper/2107.13541/) (2021)
- [Interpretable Adversarial Perturbation in Input Embedding Space for Text](https://mlsanity.com/paper/1805.02917/) (2018)
- [It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations](https://mlsanity.com/paper/2005.04364/) (2020)

## Built on by (most cited first, 3 of 3)

- [GuardReasoner: Towards Reasoning-based LLM Safeguards](https://mlsanity.com/paper/2501.18492/) (2025)
- [Confidence Elicitation: A New Attack Vector for Large Language Models](https://mlsanity.com/paper/2502.04643/) (2025)
- [DiffuseDef: Improved Robustness to Adversarial Attacks via Iterative Denoising](https://mlsanity.com/paper/2407.00248/) (2024)

## Abstract (arXiv)

Language models (LMs) are indispensable tools for natural language processing tasks, but their vulnerability to adversarial attacks remains a concern. While current research has explored adversarial training techniques, their improvements to defend against word-level attacks have been limited. In this work, we propose a novel approach called Semantic Robust Defence (SemRoDe), a Macro Adversarial Training strategy to enhance the robustness of LMs. Drawing inspiration from recent studies in the image domain, we investigate and later confirm that in a discrete data setting such as language, adversarial samples generated via word substitutions do indeed belong to an adversarial domain exhibiting a high Wasserstein distance from the base domain. Our method learns a robust representation that bridges these two domains. We hypothesize that if samples were not projected into an adversarial domain, but instead to a domain with minimal shift, it would improve attack robustness. We align the domains by incorporating a new distance-based objective. With this, our model is able to learn more generalized representations by aligning the model's high-level output features and therefore better handling unseen adversarial samples. This method can be generalized across word embeddings, even when they share minimal overlap at both vocabulary and word-substitution levels. To evaluate the effectiveness of our approach, we conduct experiments on BERT and RoBERTa models on three datasets. The results demonstrate promising state-of-the-art robustness.