# Exploring the Universal Vulnerability of Prompt-based Learning Paradigm

arXiv: https://arxiv.org/abs/2204.05239 · HTML: https://mlsanity.com/paper/2204.05239/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger](https://mlsanity.com/paper/2105.12400/) (2021)
- [Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer](https://mlsanity.com/paper/2110.07139/) (2021)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Hidden Backdoors in Human-Centric Language Models](https://mlsanity.com/paper/2105.00164/) (2021)
- [Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks](https://mlsanity.com/paper/2101.06969/) (2021)
- [CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generation](https://mlsanity.com/paper/2010.02338/) (2020)
- [Multi-granularity Textual Adversarial Attack with Behavior Cloning](https://mlsanity.com/paper/2109.04367/) (2021)
- [Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features](https://mlsanity.com/paper/1709.01189/) (2017)

## Built on by (most cited first, 30 of 57)

- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Jatmo: Prompt Injection Defense by Task-Specific Finetuning](https://mlsanity.com/paper/2312.17673/) (2023)
- [DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback](https://mlsanity.com/paper/2311.10081/) (2023)
- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models](https://mlsanity.com/paper/2305.01219/) (2023)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models](https://mlsanity.com/paper/2401.12242/) (2024)
- [A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks](https://mlsanity.com/paper/2206.08514/) (2022)
- [BadPrompt: Backdoor Attacks on Continuous Prompts](https://mlsanity.com/paper/2211.14719/) (2022)
- [Black Box Adversarial Prompting for Foundation Models](https://mlsanity.com/paper/2302.04237/) (2023)
- [Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning](https://mlsanity.com/paper/2401.05949/) (2024)
- [PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models](https://mlsanity.com/paper/2310.12439/) (2023)
- [Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review](https://mlsanity.com/paper/2309.06055/) (2023)
- [NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models](https://mlsanity.com/paper/2305.17826/) (2023)
- [Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices](https://mlsanity.com/paper/2403.12503/) (2024)
- [Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks](https://mlsanity.com/paper/2309.13256/) (2023)
- [PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning](https://mlsanity.com/paper/2410.08811/) (2024)
- [TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models](https://mlsanity.com/paper/2306.06815/) (2023)
- [Universal Prompt Optimizer for Safe Text-to-Image Generation](https://mlsanity.com/paper/2402.10882/) (2024)
- [Evaluating the Robustness of Text-to-image Diffusion Models against Real-world Attacks](https://mlsanity.com/paper/2306.13103/) (2023)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP](https://mlsanity.com/paper/2308.02122/) (2023)
- [Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2402.12168/) (2024)
- [PromptAttack: Prompt-based Attack for Language Models via Gradient Search](https://mlsanity.com/paper/2209.01882/) (2022)
- [From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing](https://mlsanity.com/paper/2410.04772/) (2024)
- [TuBA: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning](https://mlsanity.com/paper/2404.19597/) (2024)
- [Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models](https://mlsanity.com/paper/2407.11282/) (2024)
- [A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models](https://mlsanity.com/paper/2307.12980/) (2023)

## Abstract (arXiv)

Prompt-based learning paradigm bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. In this paper, we explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text. In both scenarios, we demonstrate that our triggers can totally control or severely decrease the performance of prompt-based models fine-tuned on arbitrary downstream tasks, reflecting the universal vulnerability of the prompt-based learning paradigm. Further experiments show that adversarial triggers have good transferability among language models. We also find conventional fine-tuning models are not vulnerable to adversarial triggers constructed from pre-trained language models. We conclude by proposing a potential solution to mitigate our attack methods. Code and data are publicly available at https://github.com/leix28/prompt-universal-vulnerability