mlsanityData to 7 Oct 2026

Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning

arXiv 2401.05949 ↗First posted 11 Jan 2024cs.CL cs.AI cs.CRcited by 47

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we design a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning demonstration prompts, which can make models behave in alignment with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model's generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 180B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 44 of its 73 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

44 of 73 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations16 Nov
  2. 2023Backdoor Attacks for In-Context Learning with Language Models27 Jul
  3. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models2 May
  4. 2023Training-free Lexical Backdoor Attacks on Language Models8 Feb
  5. 2022BadPrompt: Backdoor Attacks on Continuous Prompts27 Nov
  6. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  7. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May

Other references · 37

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  12. 2021MetaICL: Learning to Learn In Contextcited by 450
25 more
  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  2. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  3. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  4. 2023Poisoning Language Models During Instruction Tuningcited by 254
  5. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  6. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  7. 2022Active Example Selection for In-Context Learningcited by 195
  8. 2023Planning with Large Language Models for Code Generationcited by 179
  9. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  10. 2023Compositional Exemplars for In-context Learningcited by 143
  11. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  12. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  13. 2023Symbol tuning improves in-context learning in language modelscited by 72
  14. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  15. 2023In-context Example Selection with Influencescited by 60
  16. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  17. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  18. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  19. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  20. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  21. 2022BadHash: Invisible Backdoor Attacks against Deep Hashing with Clean Labelcited by 28
  22. 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 26
  23. 2022Certified Robustness Against Natural Language Attacks by Causal Interventioncited by 23
  24. 2021Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillationcited by 22
  25. 2023A White-Box False Positive Adversarial Attack Method on Contrastive Loss Based Offline Handwritten Signature Verification Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 20; 2026: 6.

47 papers.

Sort:

By area:

  1. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  2. 2024Safeguarding Large Language Models: A Surveycited by 73
  3. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  4. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  5. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  6. 2024Compromising Embodied Agents with Contextual Backdoor Attackscited by 29
  7. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  8. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 21
  9. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  10. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
  11. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  12. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
35 more
  1. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  2. 2025Uni-Retrieval: A Multi-Style Retrieval Framework for STEM's Educationcited by 13
  3. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  4. 2025A Survey of Attacks on Large Language Modelscited by 13
  5. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  6. 2024A Study of Backdoors in Instruction Fine-tuned Language Modelscited by 11
  7. 2025ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Modelscited by 10
  8. 2024Artwork Protection Against Neural Style Transfer Using Locally Adaptive Adversarial Color Attackcited by 9
  9. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  10. 2026Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Usecited by 9
  11. 2024Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuningcited by 8
  12. 2025Towards Robust Evaluation of STEM Education: Leveraging MLLMs in Project-Based Learningcited by 7
  13. 2025Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysiscited by 6
  14. 2025Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigationcited by 5
  15. 2024SynGhost: Invisible and Universal Task-agnostic Backdoor Attack via Syntactic Transfercited by 4
  16. 2024Neutralizing Backdoors through Information Conflicts for Large Language Modelscited by 4
  17. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMscited by 4
  18. 2025On the Robustness of Tabular Foundation Models: Test-Time Attacks and In-Context Defensescited by 3
  19. 2025Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMscited by 3
  20. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  21. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3
  22. 2024Injecting Undetectable Backdoors in Obfuscated Neural Networks and Language Modelscited by 2
  23. 2024Krait: A Backdoor Attack Against Graph Prompt Tuningcited by 2
  24. 2024Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMscited by 2
  25. 2025From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEMcited by 2
  26. 2025DUP: Detection-guided Unlearning for Backdoor Purification in Language Modelscited by 2
  27. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  28. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Modelscited by 1
  29. 2026Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggerscited by 1
  30. 2024Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimizationcited by 0
  31. 2025Robust In-Context Reinforcement Learning Under Reward Poisoning Attackscited by 0
  32. 2026Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromisecited by 0
  33. 2026Where Detectors Fail: Closing the Tail-Domain Gap with Expert-Guided Mutual Distillationcited by 0
  34. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  35. 2026OPBackdoor: Opportunistic Backdoors via Alibi-Aligned Reasoningcited by 0