mlsanityData to 7 Oct 2026

Backdoor Attacks for In-Context Learning with Language Models

arXiv 2307.14692 ↗First posted 27 Jul 2023cs.CRcited by 92

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Because state-of-the-art language models are expensive to train, most practitioners must make use of one of the few publicly available language models or language model APIs. This consolidation of trust increases the potency of backdoor attacks, where an adversary tampers with a machine learning model in order to make it perform some malicious behavior on inputs that contain a predefined backdoor trigger. We show that the in-context learning ability of large language models significantly complicates the question of developing backdoor attacks, as a successful backdoor must work against various prompting strategies and should not affect the model's general purpose capabilities. We design a new attack for eliciting targeted misclassification when language models are prompted to perform a particular target task and demonstrate the feasibility of this attack by backdooring multiple large language models ranging in size from 1.3 billion to 6 billion parameters. Finally we study defenses to mitigate the potential harms of our attack: for example, while in the white-box setting we show that fine-tuning models for as few as 500 steps suffices to remove the backdoor behavior, in the black-box setting we are unable to develop a successful defense that relies on prompt engineering alone.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. First posted on arXiv 27 Jul 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 43; 2025: 26; 2026: 14.

92 papers.

Sort:

By area:

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  4. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  5. 2023Large Language Model Alignment: A Surveycited by 241
  6. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  7. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  8. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  9. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  10. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  11. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  12. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
38 more
  1. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  2. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  3. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  4. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  5. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 61
  6. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  7. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  8. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  10. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  11. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  12. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  13. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  14. 2024Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Techniquecited by 34
  15. 2025Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Modelscited by 32
  16. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  17. 2024Compromising Embodied Agents with Contextual Backdoor Attackscited by 29
  18. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  19. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  20. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  21. 2023Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanismcited by 21
  22. 2024Token-wise Influential Training Data Retrieval for Large Language Modelscited by 19
  23. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  24. 2024Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Modelscited by 17
  25. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  26. 2025DarkMind: Latent Chain-of-Thought Backdoor in Customized LLMscited by 15
  27. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  28. 2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learningcited by 13
  29. 2024Uncertainty is Fragile: Manipulating Uncertainty in Large Language Modelscited by 13
  30. 2025Learning Safety Constraints for Large Language Modelscited by 13
  31. 2023Maatphor: Automated Variant Analysis for Prompt Injection Attackscited by 12
  32. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  33. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  34. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  35. 2024Stepwise Reasoning Error Disruption Attack of LLMscited by 8
  36. 2024Jailbreaking? One Step Is Enough!cited by 8
  37. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 8
  38. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 7

50 most cited of 92 All 92 (JSON)