mlsanityData to 7 Oct 2026

Instruction Backdoor Attacks Against Customized LLMs

arXiv 2402.09179 ↗First posted 14 Feb 2024cs.CR cs.LGcited by 50

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

The increasing demand for customized Large Language Models (LLMs) has led to the development of solutions like GPTs. These solutions facilitate tailored LLM creation via natural language prompts without coding. However, the trustworthiness of third-party custom versions of LLMs remains an essential concern. In this paper, we propose the first instruction backdoor attacks against applications integrated with untrusted customized LLMs (e.g., GPTs). Specifically, these attacks embed the backdoor into the custom version of LLMs by designing prompts with backdoor instructions, outputting the attacker's desired result when inputs contain the pre-defined triggers. Our attack includes 3 levels of attacks: word-level, syntax-level, and semantic-level, which adopt different types of triggers with progressive stealthiness. We stress that our attacks do not require fine-tuning or any modification to the backend LLMs, adhering strictly to GPTs development guidelines. We conduct extensive experiments on 6 prominent LLMs and 5 benchmark text classification datasets. The results show that our instruction backdoor attacks achieve the desired attack performance without compromising utility. Additionally, we propose two defense strategies and demonstrate their effectiveness in reducing such attacks. Our findings highlight the vulnerability and the potential risks of LLM customization such as GPTs.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 55 of its 85 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

55 of 85 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Composite Backdoor Attacks Against Large Language Models11 Oct
  2. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection31 Jul
  3. 2023Backdoor Attacks for In-Context Learning with Language Models27 Jul
  4. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models2 May
  5. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning1 Aug

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2024Mixtral of Expertscited by 1,785
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2018How To Backdoor Federated Learningcited by 745
  13. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  14. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  15. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  16. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  17. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  18. 2020Backdoor Learning: A Surveycited by 412
  19. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  20. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  21. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  22. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  23. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  24. 2023Poisoning Language Models During Instruction Tuningcited by 254
  25. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  26. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  27. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  28. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  29. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  30. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  31. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 85
  32. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  33. 2019The relationship between trust in AI and trustworthy machine learning technologiescited by 78
  34. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  35. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  36. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35
  37. 2023Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakagecited by 26
  38. 2023CancerGPT: Few-shot Drug Pair Synergy Prediction using Large Pre-trained Language Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 24; 2026: 18.

50 papers.

Sort:

By area:

  1. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  2. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  3. 2024Membership Inference Attacks Against In-Context Learningcited by 38
  4. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
  5. 2025MPMA: Preference Manipulation Attack Against Model Context Protocolcited by 31
  6. 2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 29
  7. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  8. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25
  9. 2026BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoningcited by 25
  10. 2025DarkMind: Latent Chain-of-Thought Backdoor in Customized LLMscited by 15
  11. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 15
  12. 2025A Survey of Attacks on Large Language Modelscited by 13
38 more
  1. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  2. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  3. 2025ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Modelscited by 10
  4. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  5. 2024A Survey on Recommendation Unlearning: Fundamentals, Taxonomy, Evaluation, and Open Questionscited by 7
  6. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  7. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systemscited by 7
  8. 2025DeBackdoor: A Deductive Framework for Detecting Backdoor Attacks on Deep Models with Limited Datacited by 5
  9. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMscited by 4
  10. 2025"I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairmentscited by 3
  11. 2025Pay Attention to the Triggers: Constructing Backdoors That Survive Distillationcited by 3
  12. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  13. 2025A Linear Approach to Data Poisoningcited by 2
  14. 2025Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Modelscited by 2
  15. 2025BadLingual: A Novel Lingual-Backdoor Attack against Large Language Modelscited by 1
  16. 2025The Ripple Effect: On Unforeseen Complications of Backdoor Attackscited by 1
  17. 2025A Survey: Towards Privacy and Security in Mobile Large Language Modelscited by 1
  18. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  19. 2025Stealthy Dual-Trigger Backdoors: Attacking Prompt Tuning in LM-Empowered Graph Foundation Modelscited by 1
  20. 2025SteganoBackdoor: Evading Data-Poisoning Defenses via Steganographic Backdoorscited by 1
  21. 2026Towards Poisoning Robustness Certification for Natural Language Generationcited by 1
  22. 2026PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 1
  23. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  24. 2026Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggerscited by 1
  25. 2026MetaBackdoor: Exploiting Positional Encoding as a Backdoor Attack Surface in LLMscited by 1
  26. 2026RogueMerge: Robust and Unified Attacks against LLM Model Mergingcited by 1
  27. 2024Invisible Textual Backdoor Attacks based on Dual-Triggercited by 0
  28. 2025Lethe: Purifying Backdoored Large Language Models with Knowledge Dilutioncited by 0
  29. 2025Localizing Malicious Outputs from CodeLLMcited by 0
  30. 2026Turn-Based Structural Triggers: Structure-Conditioned Backdoors in Multi-Turn LLMscited by 0
  31. 2026Trustworthy AI LLM Scalability Risk Index (LSRI): A Cybersecurity Framework Assessing Agentic-AI Security & Software Model Supply Chain Safety Boosting AI-Generated Malware Defense & Explainability Mitigating Emerging Risks of Generative AIcited by 0
  32. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Trainingcited by 0
  33. 2026Low Rank Adaptation for Adversarial Perturbationcited by 0
  34. 2026Cordyceps: Covert Control Attacks on LLMs via Data Poisoningcited by 0
  35. 2026ElasticBack: Stealthy Conditional Backdoor in LLM-Agent Skills via Coupled Trigger-Rule Optimizationcited by 0
  36. 2026Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMscited by 0
  37. 2026SpecGuard: Inference-Time Backdoor Detection For Freecited by 0
  38. 2026Silent Sabotage: Internal State Triggered Backdoor Attacks on LLM-Powered Robotic Systemscited by 0