# Robust Encodings: A Framework for Combating Adversarial Typos

arXiv: https://arxiv.org/abs/2005.01229 · HTML: https://mlsanity.com/paper/2005.01229/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 30 of 58)

- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [VALUE: Understanding Dialect Disparity in NLU](https://mlsanity.com/paper/2204.03031/) (2022)
- [SemAttack: Natural Textual Attacks via Different Semantic Spaces](https://mlsanity.com/paper/2205.01287/) (2022)
- [LM-Critic: Language Models for Unsupervised Grammatical Error Correction](https://mlsanity.com/paper/2109.06822/) (2021)
- [Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimation](https://mlsanity.com/paper/2203.01677/) (2022)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [Achieving Model Robustness through Discrete Adversarial Training](https://mlsanity.com/paper/2104.05062/) (2021)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Phrase-level Textual Adversarial Attack with Label Preservation](https://mlsanity.com/paper/2205.10710/) (2022)
- [P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts](https://mlsanity.com/paper/2110.07280/) (2021)
- [Understanding the Generalization Benefit of Model Invariance from a Data Perspective](https://mlsanity.com/paper/2111.05529/) (2021)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [Multi-granularity Textual Adversarial Attack with Behavior Cloning](https://mlsanity.com/paper/2109.04367/) (2021)
- [BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks](https://mlsanity.com/paper/2106.01452/) (2021)
- [On the Robustness of Reading Comprehension Models to Entity Renaming](https://mlsanity.com/paper/2110.08555/) (2021)
- [Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation](https://mlsanity.com/paper/2107.11252/) (2021)
- [On Robust Prefix-Tuning for Text Classification](https://mlsanity.com/paper/2203.10378/) (2022)
- [AEON: A Method for Automatic Evaluation of NLP Test Cases](https://mlsanity.com/paper/2205.06439/) (2022)
- [Adversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretation](https://mlsanity.com/paper/2203.12709/) (2022)
- [Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing](https://mlsanity.com/paper/2310.13855/) (2023)
- [A Differentiable Language Model Adversarial Attack on Text Classifiers](https://mlsanity.com/paper/2107.11275/) (2021)
- [Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics](https://mlsanity.com/paper/2112.08321/) (2021)
- [Empirical Error Modeling Improves Robustness of Noisy Neural Sequence Labeling](https://mlsanity.com/paper/2105.11872/) (2021)
- [Detecting Textual Adversarial Examples through Randomized Substitution and Vote](https://mlsanity.com/paper/2109.05698/) (2021)
- [TextShield: Beyond Successfully Detecting Adversarial Sentences in Text Classification](https://mlsanity.com/paper/2302.02023/) (2023)
- [ER-Test: Evaluating Explanation Regularization Methods for Language Models](https://mlsanity.com/paper/2205.12542/) (2022)
- [Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models](https://mlsanity.com/paper/2109.05793/) (2021)
- [Data-Driven Mitigation of Adversarial Text Perturbation](https://mlsanity.com/paper/2202.09483/) (2022)

## Abstract (arXiv)

Despite excellent performance on many tasks, NLP systems are easily fooled by small adversarial perturbations of inputs. Existing procedures to defend against such perturbations are either (i) heuristic in nature and susceptible to stronger attacks or (ii) provide guaranteed robustness to worst-case attacks, but are incompatible with state-of-the-art models like BERT. In this work, we introduce robust encodings (RobEn): a simple framework that confers guaranteed robustness, without making compromises on model architecture. The core component of RobEn is an encoding function, which maps sentences to a smaller, discrete space of encodings. Systems using these encodings as a bottleneck confer guaranteed robustness with standard training, and the same encodings can be used across multiple tasks. We identify two desiderata to construct robust encoding functions: perturbations of a sentence should map to a small set of encodings (stability), and models using encodings should still perform well (fidelity). We instantiate RobEn to defend against a large family of adversarial typos. Across six tasks from GLUE, our instantiation of RobEn paired with BERT achieves an average robust accuracy of 71.3% against all adversarial typos in the family considered, while previous work using a typo-corrector achieves only 35.3% accuracy against a simple greedy attack.