mlsanityData to 7 Oct 2026

ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks

arXiv 2507.01321 ↗First posted 2 Jul 2025cs.LG cs.AI cs.CRcited by 9

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces a critical vulnerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poisoning a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstrations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability is dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Our method encourages LLMs to select clean demonstrations during the ICL phase by leveraging confidence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive experiments across multiple LLMs and tasks demonstrate that our method achieves state-of-the-art defense effectiveness, significantly outperforming existing approaches (+26.02% on average). Furthermore, our method exhibits exceptional adaptability and defensive performance even for closed-source models (e.g., GPT-4).

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 54 of its 66 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

54 of 66 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Adversarial Backdoor Defense in CLIP24 Sep
  2. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift27 Jun
  3. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models24 Jun
  4. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning24 Mar
  5. 2024BadEdit: Backdooring large language models by model editing20 Mar
  6. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning11 Jan
  7. 2023Pre-trained Trojan Attacks for Visual Recognition23 Dec
  8. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning20 Nov
  9. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations16 Nov
  10. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection31 Jul
  11. 2023Backdoor Attacks for In-Context Learning with Language Models27 Jul
  12. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models24 May
1 more
  1. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
29 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  7. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  8. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  9. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  10. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  12. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  13. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  14. 2023Compositional Exemplars for In-context Learningcited by 143
  15. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  16. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  17. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  18. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  19. 2024Language Models for Text Classification: Is In-Context Learning Enough?cited by 39
  20. 2023In-Context Learning for Text Classification with Many Labelscited by 35
  21. 2024Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detectioncited by 34
  22. 2024Compromising Embodied Agents with Contextual Backdoor Attackscited by 29
  23. 2024Towards Robust Physical-world Backdoor Attacks on Lane Detectioncited by 25
  24. 2023RobustMQ: Benchmarking Robustness of Quantized Modelscited by 19
  25. 2024Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultationcited by 17
  26. 2022Universal Backdoor Attacks Detection via Adaptive Adversarial Probecited by 16
  27. 2024Wordflow: Social Prompt Engineering for Large Language Modelscited by 9
  28. 2024What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysiscited by 7
  29. 2024Fine-grained Controllable Text Generation through In-context Learning with Feedbackcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMscited by 1
  2. 2026Distillation Traps and Guards: A Calibration Knob for LLM Distillabilitycited by 1
  3. 2025Towards Size-invariant Salient Object Detection: A Generic Evaluation and Optimization Approachcited by 0
  4. 2025SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Modelscited by 0
  5. 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Modelscited by 0
  6. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
  7. 2026Breadcrumbing Search Agentscited by 0
  8. 2026Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Impact, Detection, and Mitigationcited by 0
  9. 2026Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Model Dependence and Evaluation Reliabilitycited by 0