# Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

arXiv: https://arxiv.org/abs/1803.01128 · HTML: https://mlsanity.com/paper/1803.01128/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2018.

## Built on by (most cited first, 30 of 94)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [RADAR: Robust AI-Text Detection via Adversarial Learning](https://mlsanity.com/paper/2307.03838/) (2023)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Towards Improving Adversarial Training of NLP Models](https://mlsanity.com/paper/2109.00544/) (2021)
- [Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures](https://mlsanity.com/paper/2112.05224/) (2021)
- [Certified Robustness to Text Adversarial Attacks by Randomized [MASK]](https://mlsanity.com/paper/2105.03743/) (2021)
- [Adversarial Attacks On Multi-Agent Communication](https://mlsanity.com/paper/2101.06560/) (2021)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbalanced Data](https://mlsanity.com/paper/2101.08030/) (2021)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [SemAttack: Natural Textual Attacks via Different Semantic Spaces](https://mlsanity.com/paper/2205.01287/) (2022)
- [Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation](https://mlsanity.com/paper/2402.11493/) (2024)
- [Dual-Key Multimodal Backdoors for Visual Question Answering](https://mlsanity.com/paper/2112.07668/) (2021)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models](https://mlsanity.com/paper/2105.11136/) (2021)
- [A Review of Adversarial Attack and Defense for Classification Methods](https://mlsanity.com/paper/2111.09961/) (2021)
- [Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitution](https://mlsanity.com/paper/2210.17004/) (2022)
- [A Survey on Out-of-Distribution Evaluation of Neural NLP Models](https://mlsanity.com/paper/2306.15261/) (2023)
- [Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Models](https://mlsanity.com/paper/2105.14813/) (2021)
- [Efficient Combinatorial Optimization for Word-level Adversarial Textual Attack](https://mlsanity.com/paper/2109.02229/) (2021)
- [Reverse Engineering of Imperceptible Adversarial Image Perturbations](https://mlsanity.com/paper/2203.14145/) (2022)
- [The King is Naked: on the Notion of Robustness for Natural Language Processing](https://mlsanity.com/paper/2112.07605/) (2021)
- [LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Models](https://mlsanity.com/paper/2210.03696/) (2022)
- [HQA-Attack: Toward High Quality Black-Box Hard-Label Adversarial Attack on Text](https://mlsanity.com/paper/2402.01806/) (2024)
- [On Robustness of Neural Semantic Parsers](https://mlsanity.com/paper/2102.01563/) (2021)
- [White-Box Multi-Objective Adversarial Attack on Dialogue Generation](https://mlsanity.com/paper/2305.03655/) (2023)
- [Dynamic Transformers Provide a False Sense of Efficiency](https://mlsanity.com/paper/2305.12228/) (2023)
- [Towards Building a Robust Toxicity Predictor](https://mlsanity.com/paper/2404.08690/) (2024)

## Abstract (arXiv)

Crafting adversarial examples has become an important technique to evaluate the robustness of deep neural networks (DNNs). However, most existing works focus on attacking the image classification problem since its input space is continuous and output space is finite. In this paper, we study the much more challenging problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models, whose inputs are discrete text strings and outputs have an almost infinite number of possibilities. To address the challenges caused by the discrete input space, we propose a projected gradient method combined with group lasso and gradient regularization. To handle the almost infinite output space, we design some novel loss functions to conduct non-overlapping attack and targeted keyword attack. We apply our algorithm to machine translation and text summarization tasks, and verify the effectiveness of the proposed algorithm: by changing less than 3 words, we can make seq2seq model to produce desired outputs with high success rates. On the other hand, we recognize that, compared with the well-evaluated CNN-based classifiers, seq2seq models are intrinsically more robust to adversarial attacks.