# Using BERT Encoding to Tackle the Mad-lib Attack in SMS Spam Detection

arXiv: https://arxiv.org/abs/2107.06400 · HTML: https://mlsanity.com/paper/2107.06400/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [Google's Cloud Vision API Is Not Robust To Noise](https://mlsanity.com/paper/1704.05051/) (2017)
- [No Rumours Please! A Multi-Indic-Lingual Approach for COVID Fake-Tweet Detection](https://mlsanity.com/paper/2010.06906/) (2020)
- [CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails](https://mlsanity.com/paper/2010.03484/) (2020)
- [Shielding Google's language toxicity model against adversarial attacks](https://mlsanity.com/paper/1801.01828/) (2018)

## Built on by (most cited first, 3 of 3)

- [Zero-Shot Spam Email Classification Using Pre-trained Large Language Models](https://mlsanity.com/paper/2405.15936/) (2024)
- [Bengali Fake Reviews: A Benchmark Dataset and Detection System](https://mlsanity.com/paper/2308.01987/) (2023)
- [Conditional Semi-Supervised Data Augmentation for Spam Message Detection with Low Resource Data](https://mlsanity.com/paper/2407.04990/) (2024)

## Abstract (arXiv)

One of the stratagems used to deceive spam filters is to substitute vocables with synonyms or similar words that turn the message unrecognisable by the detection algorithms. In this paper we investigate whether the recent development of language models sensitive to the semantics and context of words, such as Google's BERT, may be useful to overcome this adversarial attack (called "Mad-lib" as per the word substitution game). Using a dataset of 5572 SMS spam messages, we first established a baseline of detection performance using widely known document representation models (BoW and TFIDF) and the novel BERT model, coupled with a variety of classification algorithms (Decision Tree, kNN, SVM, Logistic Regression, Naive Bayes, Multilayer Perceptron). Then, we built a thesaurus of the vocabulary contained in these messages, and set up a Mad-lib attack experiment in which we modified each message of a held out subset of data (not used in the baseline experiment) with different rates of substitution of original words with synonyms from the thesaurus. Lastly, we evaluated the detection performance of the three representation models (BoW, TFIDF and BERT) coupled with the best classifier from the baseline experiment (SVM). We found that the classic models achieved a 94% Balanced Accuracy (BA) in the original dataset, whereas the BERT model obtained 96%. On the other hand, the Mad-lib attack experiment showed that BERT encodings manage to maintain a similar BA performance of 96% with an average substitution rate of 1.82 words per message, and 95% with 3.34 words substituted per message. In contrast, the BA performance of the BoW and TFIDF encoders dropped to chance. These results hint at the potential advantage of BERT models to combat these type of ingenious attacks, offsetting to some extent for the inappropriate use of semantic relationships in language.