# SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions

arXiv: https://arxiv.org/abs/2005.14424 · HTML: https://mlsanity.com/paper/2005.14424/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2020.

## Built on by (most cited first, 30 of 73)

- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing](https://mlsanity.com/paper/2402.16192/) (2024)
- [Certified Robustness to Text Adversarial Attacks by Randomized [MASK]](https://mlsanity.com/paper/2105.03743/) (2021)
- [Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution](https://mlsanity.com/paper/2108.12777/) (2021)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks](https://mlsanity.com/paper/2307.16630/) (2023)
- [Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrifice](https://mlsanity.com/paper/2105.14553/) (2021)
- [Certified Robustness Against Natural Language Attacks by Causal Intervention](https://mlsanity.com/paper/2205.12331/) (2022)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Certified Robustness for Large Language Models with Self-Denoising](https://mlsanity.com/paper/2307.07171/) (2023)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification](https://mlsanity.com/paper/2106.10826/) (2021)
- [Textual Manifold-based Defense Against Natural Language Adversarial Examples](https://mlsanity.com/paper/2211.02878/) (2022)
- [Certified Robustness to Programmable Transformations in LSTMs](https://mlsanity.com/paper/2102.07818/) (2021)
- [On the Robustness of Reading Comprehension Models to Entity Renaming](https://mlsanity.com/paper/2110.08555/) (2021)
- [RS-Del: Edit Distance Robustness Certificates for Sequence Classifiers via Randomized Deletion](https://mlsanity.com/paper/2302.01757/) (2023)
- [NLP Verification: Towards a General Methodology for Certifying Robustness](https://mlsanity.com/paper/2403.10144/) (2024)
- [Advancing the Robustness of Large Language Models through Self-Denoised Smoothing](https://mlsanity.com/paper/2404.12274/) (2024)
- [On Robust Prefix-Tuning for Text Classification](https://mlsanity.com/paper/2203.10378/) (2022)
- [Adversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretation](https://mlsanity.com/paper/2203.12709/) (2022)
- [Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2201.00768/) (2022)
- [Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications](https://mlsanity.com/paper/2305.06522/) (2023)
- [Impact of Adversarial Training on Robustness and Generalizability of Language Models](https://mlsanity.com/paper/2211.05523/) (2022)
- [Less is More: Understanding Word-level Textual Adversarial Attack via n-gram Frequency Descend](https://mlsanity.com/paper/2302.02568/) (2023)
- [Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Text](https://mlsanity.com/paper/2305.16444/) (2023)
- [Integer-arithmetic-only Certified Robustness for Quantized Neural Networks](https://mlsanity.com/paper/2108.09413/) (2021)

## Abstract (arXiv)

State-of-the-art NLP models can often be fooled by human-unaware transformations such as synonymous word substitution. For security reasons, it is of critical importance to develop models with certified robustness that can provably guarantee that the prediction is can not be altered by any possible synonymous word substitution. In this work, we propose a certified robust method based on a new randomized smoothing technique, which constructs a stochastic ensemble by applying random word substitutions on the input sentences, and leverage the statistical properties of the ensemble to provably certify the robustness. Our method is simple and structure-free in that it only requires the black-box queries of the model outputs, and hence can be applied to any pre-trained models (such as BERT) and any types of models (world-level or subword-level). Our method significantly outperforms recent state-of-the-art methods for certified robustness on both IMDB and Amazon text classification tasks. To the best of our knowledge, we are the first work to achieve certified robustness on large systems such as BERT with practically meaningful certified accuracy.