# Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformers

arXiv: https://arxiv.org/abs/2409.04142 · HTML: https://mlsanity.com/paper/2409.04142/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Deep Retinex Decomposition for Low-Light Enhancement](https://mlsanity.com/paper/1808.04560/) (2018)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Multi-Scale Progressive Fusion Network for Single Image Deraining](https://mlsanity.com/paper/2003.10985/) (2020)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [Backdoor Attacks for In-Context Learning with Language Models](https://mlsanity.com/paper/2307.14692/) (2023)
- [Hidden Backdoors in Human-Centric Language Models](https://mlsanity.com/paper/2105.00164/) (2021)
- [UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes](https://mlsanity.com/paper/2205.10337/) (2022)
- [Composite Backdoor Attacks Against Large Language Models](https://mlsanity.com/paper/2310.07676/) (2023)
- [Handcrafted Backdoors in Deep Neural Networks](https://mlsanity.com/paper/2106.04690/) (2021)
- [Indoor Semantic Segmentation using depth information](https://mlsanity.com/paper/1301.3572/) (2013)
- [Backdoor Attacks against Transfer Learning with Pre-trained Deep Learning Models](https://mlsanity.com/paper/2001.03274/) (2020)
- [Can You Hear It? Backdoor Attacks via Ultrasonic Triggers](https://mlsanity.com/paper/2107.14569/) (2021)
- [Explainability-based Backdoor Attacks Against Graph Neural Networks](https://mlsanity.com/paper/2104.03674/) (2021)
- [TrojViT: Trojan Insertion in Vision Transformers](https://mlsanity.com/paper/2208.13049/) (2022)
- [Sneaky Spikes: Uncovering Stealthy Backdoor Attacks in Spiking Neural Networks with Neuromorphic Data](https://mlsanity.com/paper/2302.06279/) (2023)
- [Backdoor Attacks on Vision Transformers](https://mlsanity.com/paper/2206.08477/) (2022)
- [When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models](https://mlsanity.com/paper/2404.08634/) (2024)
- [SoK: A Systematic Evaluation of Backdoor Trigger Characteristics in Image Classification](https://mlsanity.com/paper/2302.01740/) (2023)

## Built on by (most cited first, 2 of 2)

- [On the Robustness of Tabular Foundation Models: Test-Time Attacks and In-Context Defenses](https://mlsanity.com/paper/2506.02978/) (2025)
- [Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defenses](https://mlsanity.com/paper/2510.08016/) (2025)

## Abstract (arXiv)

Due to the high cost of training, large model (LM) practitioners commonly use pretrained models downloaded from untrusted sources, which could lead to owning compromised models. In-context learning is the ability of LMs to perform multiple tasks depending on the prompt or context. This can enable new attacks, such as backdoor attacks with dynamic behavior depending on how models are prompted. In this paper, we leverage the ability of vision transformers (ViTs) to perform different tasks depending on the prompts. Then, through data poisoning, we investigate two new threats: i) task-specific backdoors where the attacker chooses a target task to attack, and only the selected task is compromised at test time under the presence of the trigger. At the same time, any other task is not affected, even if prompted with the trigger. We succeeded in attacking every tested model, achieving up to 89.90% degradation on the target task. ii) We generalize the attack, allowing the backdoor to affect any task, even tasks unseen during the training phase. Our attack was successful on every tested model, achieving a maximum of 13× degradation. Finally, we investigate the robustness of prompts and fine-tuning as techniques for removing the backdoors from the model. We found that these methods fall short and, in the best case, reduce the degradation from 89.90% to 73.46%.