# BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks

arXiv: https://arxiv.org/abs/2106.01452 · HTML: https://mlsanity.com/paper/2106.01452/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Combating Adversarial Misspellings with Robust Word Recognition](https://mlsanity.com/paper/1905.11268/) (2019)
- [Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems](https://mlsanity.com/paper/1903.11508/) (2019)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [OpenAttack: An Open-source Textual Adversarial Attack Toolkit](https://mlsanity.com/paper/2009.09191/) (2020)
- [Robust Encodings: A Framework for Combating Adversarial Typos](https://mlsanity.com/paper/2005.01229/) (2020)
- [Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification](https://mlsanity.com/paper/1909.03084/) (2019)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [Word-level Textual Adversarial Attacking as Combinatorial Optimization](https://mlsanity.com/paper/1910.12196/) (2019)
- [From Hero to Z'eroe: A Benchmark of Low-Level Adversarial Attacks](https://mlsanity.com/paper/2010.05648/) (2020)
- [Word Shape Matters: Robust Machine Translation with Visual Embedding](https://mlsanity.com/paper/2010.09997/) (2020)
- [Token-Modification Adversarial Attacks for Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.00676/) (2021)
- [Misspelling Correction with Pre-trained Contextual Language Model](https://mlsanity.com/paper/2101.03204/) (2021)

## Built on by (most cited first, 17 of 17)

- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimation](https://mlsanity.com/paper/2203.01677/) (2022)
- [Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors](https://mlsanity.com/paper/2110.04399/) (2021)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [What do we Really Know about State of the Art NER?](https://mlsanity.com/paper/2205.00034/) (2022)
- [Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?](https://mlsanity.com/paper/2209.02317/) (2022)
- [DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2404.14890/) (2024)
- [CitySpec with Shield: A Secure Intelligent Assistant for Requirement Formalization](https://mlsanity.com/paper/2302.09665/) (2023)
- [Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning](https://mlsanity.com/paper/2403.07078/) (2024)
- [Don't sweat the small stuff, classify the rest: Sample Shielding to protect text classifiers against adversarial attacks](https://mlsanity.com/paper/2205.01714/) (2022)
- [The Best Defense is Attack: Repairing Semantics in Textual Adversarial Examples](https://mlsanity.com/paper/2305.04067/) (2023)
- [FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation](https://mlsanity.com/paper/2504.17311/) (2025)
- [Compressed models are NOT miniature versions of large models](https://mlsanity.com/paper/2407.13174/) (2024)
- [BitAbuse: A Dataset of Visually Perturbed Texts for Defending Phishing Attacks](https://mlsanity.com/paper/2502.05225/) (2025)
- [NLP Security and Ethics, in the Wild](https://mlsanity.com/paper/2504.06669/) (2025)

## Abstract (arXiv)

Adversarial attacks expose important blind spots of deep learning systems. While word- and sentence-level attack scenarios mostly deal with finding semantic paraphrases of the input that fool NLP models, character-level attacks typically insert typos into the input stream. It is commonly thought that these are easier to defend via spelling correction modules. In this work, we show that both a standard spellchecker and the approach of Pruthi et al. (2019), which trains to defend against insertions, deletions and swaps, perform poorly on the character-level benchmark recently proposed in Eger and Benz (2020) which includes more challenging attacks such as visual and phonetic perturbations and missing word segmentations. In contrast, we show that an untrained iterative approach which combines context-independent character-level information with context-dependent information from BERT's masked language modeling can perform on par with human crowd-workers from Amazon Mechanical Turk (AMT) supervised via 3-shot learning.