# On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models

arXiv: https://arxiv.org/abs/1903.06620 · HTML: https://mlsanity.com/paper/1903.06620/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2019.

## Built on by (most cited first, 30 of 56)

- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding](https://mlsanity.com/paper/2107.00440/) (2021)
- [Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics](https://mlsanity.com/paper/2109.13986/) (2021)
- [Reliability Testing for Natural Language Processing Systems](https://mlsanity.com/paper/2105.02590/) (2021)
- [Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots](https://mlsanity.com/paper/2103.09593/) (2021)
- [Certified Robustness to Programmable Transformations in LSTMs](https://mlsanity.com/paper/2102.07818/) (2021)
- [How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks](https://mlsanity.com/paper/2305.15587/) (2023)
- [AEON: A Method for Automatic Evaluation of NLP Test Cases](https://mlsanity.com/paper/2205.06439/) (2022)
- [On Robustness of Neural Semantic Parsers](https://mlsanity.com/paper/2102.01563/) (2021)
- [White-Box Multi-Objective Adversarial Attack on Dialogue Generation](https://mlsanity.com/paper/2305.03655/) (2023)
- [Token-Modification Adversarial Attacks for Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.00676/) (2021)
- [TransFool: An Adversarial Attack against Neural Machine Translation Models](https://mlsanity.com/paper/2302.00944/) (2023)
- [Robustness of Large Language Models Against Adversarial Attacks](https://mlsanity.com/paper/2412.17011/) (2024)
- [Unifying Model Explainability and Robustness for Joint Text Classification and Rationale Extraction](https://mlsanity.com/paper/2112.10424/) (2021)
- [When and How to Fool Explainable Models (and Humans) with Adversarial Examples](https://mlsanity.com/paper/2107.01943/) (2021)
- [Targeted Adversarial Attacks against Neural Machine Translation](https://mlsanity.com/paper/2303.01068/) (2023)
- [Consistency Training with Virtual Adversarial Discrete Perturbation](https://mlsanity.com/paper/2104.07284/) (2021)
- [CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation](https://mlsanity.com/paper/2305.17267/) (2023)
- [An Empirical Study of In-context Learning in LLMs for Machine Translation](https://mlsanity.com/paper/2401.12097/) (2024)
- [Desiderata for the Context Use of Question Answering Systems](https://mlsanity.com/paper/2401.18001/) (2024)
- [Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models](https://mlsanity.com/paper/2109.05793/) (2021)
- [Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks](https://mlsanity.com/paper/2310.16955/) (2023)
- [Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets](https://mlsanity.com/paper/2403.20056/) (2024)
- [MINIMAL: Mining Models for Data Free Universal Adversarial Triggers](https://mlsanity.com/paper/2109.12406/) (2021)
- [Adversarial Attacks and Defenses for Social Network Text Processing Applications: Techniques, Challenges and Future Research Directions](https://mlsanity.com/paper/2110.13980/) (2021)
- [Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation](https://mlsanity.com/paper/2204.08689/) (2022)
- [Frauds Bargain Attack: Generating Adversarial Text Samples via Word Manipulation Process](https://mlsanity.com/paper/2303.01234/) (2023)
- [A Classification-Guided Approach for Adversarial Attacks against Neural Machine Translation](https://mlsanity.com/paper/2308.15246/) (2023)
- [Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation](https://mlsanity.com/paper/2210.12367/) (2022)
- [Enhancing Robustness of AI Offensive Code Generators via Data Augmentation](https://mlsanity.com/paper/2306.05079/) (2023)

## Abstract (arXiv)

Adversarial examples --- perturbations to the input of a model that elicit large changes in the output --- have been shown to be an effective way of assessing the robustness of sequence-to-sequence (seq2seq) models. However, these perturbations only indicate weaknesses in the model if they do not change the input so significantly that it legitimately results in changes in the expected output. This fact has largely been ignored in the evaluations of the growing body of related literature. Using the example of untargeted attacks on machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models that takes the semantic equivalence of the pre- and post-perturbation input into account. Using this framework, we demonstrate that existing methods may not preserve meaning in general, breaking the aforementioned assumption that source side perturbations should not result in changes in the expected output. We further use this framework to demonstrate that adding additional constraints on attacks allows for adversarial perturbations that are more meaning-preserving, but nonetheless largely change the output sequence. Finally, we show that performing untargeted adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness, without hurting test performance. A toolkit implementing our evaluation framework is released at https://github.com/pmichel31415/teapot-nlp.