mlsanityData to 14 Oct 2026

Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations

arXiv 2311.09763 ↗First posted 16 Nov 2023cs.CLcited by 21

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Existing studies in backdoor defense have predominantly focused on the training phase, overlooking the critical aspect of testing time defense. This gap becomes pronounced in the context of LLMs deployed as Web Services, which typically offer only black-box access, rendering training-time defenses impractical. To bridge this gap, this study critically examines the use of demonstrations as a defense mechanism against backdoor attacks in black-box LLMs. We retrieve task-relevant demonstrations from a clean data pool and integrate them with user queries during testing. This approach does not necessitate modifications or tuning of the model, nor does it require insight into the model's internal architecture. The alignment properties inherent in in-context learning play a pivotal role in mitigating the impact of backdoor triggers, effectively recalibrating the behavior of compromised models. Our experimental analysis demonstrates that this method robustly defends against both instance-level and instruction-level backdoor attacks, outperforming existing defense baselines across most evaluation scenarios.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 49 of its 66 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

49 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Defending against Insertion-based Textual Backdoor Attacks via Attribution3 May
  2. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models18 Oct
  3. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injection25 May
  4. 2022WeDef: Weakly Supervised Backdoor Defense for Text Classification24 May
  5. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  6. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  7. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  8. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
29 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  5. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  11. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  13. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  14. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  15. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  16. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  17. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  18. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  19. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  20. 2022Prompting GPT-3 To Be Reliablecited by 260
  21. 2019A backdoor attack against LSTM-based text classification systemscited by 227
  22. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
  23. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  24. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  25. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  26. 2020Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identificationcited by 86
  27. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  28. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 27
  29. 2023Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlationcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 9; 2026: 5.

21 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 53
  4. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  5. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 27
  6. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 18
  7. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 16
  8. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 15
  9. 2024Rethinking Backdoor Detection Evaluation for Language Modelscited by 9
  10. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  11. 2025Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendationcited by 7
  12. 2025Mechanistic Anomaly Detection for "Quirky" Language Modelscited by 5
9 more
  1. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  2. 2025Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMscited by 4
  3. 2025Sharing State Between Prompts and Programscited by 2
  4. 2025Lethe: Purifying Backdoored Large Language Models with Knowledge Dilutioncited by 1
  5. 2026Triaging Threats to Specialized Guardrailscited by 1
  6. 2026DebugLM: Learning Traceable Training Data Provenance for LLMscited by 0
  7. 2026Defusing the Trigger: Tail-Risk-Informed Attention Rebalancing for LLM Backdoor Mitigationcited by 0
  8. 2026Curvature-Guided Module Localization for Low-Rank Detoxification of Backdoored Large Language Modelscited by 0
  9. 2026An Empirical Study of Output-to-Input Loops for Black-Box Backdoor Detection in Fine-Tuned Open-Weight LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
  2. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models
  3. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs
  4. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
  5. 2023A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networks
  6. 2023Backdoor Attacks for In-Context Learning with Language Models
  7. 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs
  8. 2024Transferring Backdoors between Large Language Models by Knowledge Distillation