# TrojFSP: Trojan Insertion in Few-shot Prompt Tuning

arXiv: https://arxiv.org/abs/2312.10467 · HTML: https://mlsanity.com/paper/2312.10467/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger](https://mlsanity.com/paper/2105.12400/) (2021)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models](https://mlsanity.com/paper/2110.07831/) (2021)
- [Template-free Prompt Tuning for Few-shot NER](https://mlsanity.com/paper/2109.13532/) (2021)
- [BadPrompt: Backdoor Attacks on Continuous Prompts](https://mlsanity.com/paper/2211.14719/) (2022)
- [Detecting Backdoors in Pre-trained Encoders](https://mlsanity.com/paper/2303.15180/) (2023)
- [NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models](https://mlsanity.com/paper/2305.17826/) (2023)
- [TrojText: Test-time Invisible Textual Trojan Insertion](https://mlsanity.com/paper/2303.02242/) (2023)
- [Backdoor Learning on Sequence to Sequence Models](https://mlsanity.com/paper/2305.02424/) (2023)
- [PromptAttack: Prompt-based Attack for Language Models via Gradient Search](https://mlsanity.com/paper/2209.01882/) (2022)
- [ESTAS: Effective and Stable Trojan Attacks in Self-supervised Encoders with One Target Unlabelled Sample](https://mlsanity.com/paper/2211.10908/) (2022)
- [PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts](https://mlsanity.com/paper/2306.04535/) (2023)

## Built on by (most cited first, 9 of 9)

- [BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models](https://mlsanity.com/paper/2406.00083/) (2024)
- [A Systematic Review of Poisoning Attacks Against Large Language Models](https://mlsanity.com/paper/2506.06518/) (2025)
- [Jailbreaking LLMs with Arabic Transliteration and Arabizi](https://mlsanity.com/paper/2406.18725/) (2024)
- [Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Texts](https://mlsanity.com/paper/2509.07755/) (2025)
- [BadFair: Backdoored Fairness Attacks with Group-conditioned Triggers](https://mlsanity.com/paper/2410.17492/) (2024)
- [Evaluating the Robustness and Accuracy of Text Watermarking Under Real-World Cross-Lingual Manipulations](https://mlsanity.com/paper/2502.16699/) (2025)
- [CR-UTP: Certified Robustness against Universal Text Perturbations on Large Language Models](https://mlsanity.com/paper/2406.01873/) (2024)
- [MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems](https://mlsanity.com/paper/2504.17070/) (2025)
- [REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs](https://mlsanity.com/paper/2606.23892/) (2026)

## Abstract (arXiv)

Prompt tuning is one of the most effective solutions to adapting a fixed pre-trained language model (PLM) for various downstream tasks, especially with only a few input samples. However, the security issues, e.g., Trojan attacks, of prompt tuning on a few data samples are not well-studied. Transferring established data poisoning attacks directly to few-shot prompt tuning presents multiple challenges. One significant issue is the textitpoisoned imbalance issue, where non-target class samples are added to the target class, resulting in a greater number of target-class samples compared to non-target class. While this issue is not critical in regular tuning, it significantly hampers the few-shot prompt tuning, making it difficult to simultaneously achieve a high attack success rate (ASR) and maintain clean data accuracy (CDA). Additionally, few-shot prompting is prone to overfitting in terms of both ASR and CDA. In this paper, we introduce textitTrojFSP, a method designed to address the challenges. To solve the poisoned imbalance issue, we develop a textitTarget-Class Shrink (TC-Shrink) technique, which aims to equalize the number of poisoning samples. To combat overfitting, we employ a textitSelective Token Poisoning technique to boost attack performance. Furthermore, we introduce a textitTrojan-Trigger Attention objective function to amplify the attention of the poisoned trojan prompt on triggers. Experiments show that our TrojFSP achieves an ASR of over 99% while maintaining negligible decreases in CDA across various PLMs and datasets.