# PromptAttack: Prompt-based Attack for Language Models via Gradient Search

arXiv: https://arxiv.org/abs/2209.01882 · HTML: https://mlsanity.com/paper/2209.01882/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Crafting Adversarial Input Sequences for Recurrent Neural Networks](https://mlsanity.com/paper/1604.08275/) (2016)
- [Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification](https://mlsanity.com/paper/2010.13641/) (2020)
- [Exploring the Universal Vulnerability of Prompt-based Learning Paradigm](https://mlsanity.com/paper/2204.05239/) (2022)

## Built on by (most cited first, 15 of 15)

- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models](https://mlsanity.com/paper/2306.06815/) (2023)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [Mutual-modality Adversarial Attack with Semantic Perturbation](https://mlsanity.com/paper/2312.12768/) (2023)
- [Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks](https://mlsanity.com/paper/2312.04748/) (2023)
- [TrojFSP: Trojan Insertion in Few-shot Prompt Tuning](https://mlsanity.com/paper/2312.10467/) (2023)
- [Turning Generative Models Degenerate: The Power of Data Poisoning Attacks](https://mlsanity.com/paper/2407.12281/) (2024)
- [In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models](https://mlsanity.com/paper/2409.15454/) (2024)
- [Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations](https://mlsanity.com/paper/2410.09097/) (2024)
- [Adversarial Attacks against Neural Ranking Models via In-Context Learning](https://mlsanity.com/paper/2508.15283/) (2025)
- [COVER: A Heuristic Greedy Adversarial Attack on Prompt-based Learning in Language Models](https://mlsanity.com/paper/2306.05659/) (2023)
- [TARGET: Template-Transferable Backdoor Attack Against Prompt-based NLP Models via GPT4](https://mlsanity.com/paper/2311.17429/) (2023)
- [textitLinkPrompt: Natural and Universal Adversarial Attacks on Prompt-based Language Models](https://mlsanity.com/paper/2403.16432/) (2024)
- [Towards Universal and Black-Box Query-Response Only Attack on LLMs with QROA](https://mlsanity.com/paper/2406.02044/) (2024)
- [Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?](https://mlsanity.com/paper/2408.02651/) (2024)

## Abstract (arXiv)

As the pre-trained language models (PLMs) continue to grow, so do the hardware and data requirements for fine-tuning PLMs. Therefore, the researchers have come up with a lighter method called textitPrompt Learning. However, during the investigations, we observe that the prompt learning methods are vulnerable and can easily be attacked by some illegally constructed prompts, resulting in classification errors, and serious security problems for PLMs. Most of the current research ignores the security issue of prompt-based methods. Therefore, in this paper, we propose a malicious prompt template construction method (PromptAttack) to probe the security performance of PLMs. Several unfriendly template construction approaches are investigated to guide the model to misclassify the task. Extensive experiments on three datasets and three PLMs prove the effectiveness of our proposed approach PromptAttack. We also conduct experiments to verify that our method is applicable in few-shot scenarios.