mlsanityData to 7 Oct 2026

Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models

arXiv 2406.05948 ↗First posted 10 Jun 2024cs.CR cs.AIcited by 17

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Large Language Models (LLMs), especially those accessed via APIs, have demonstrated impressive capabilities across various domains. However, users without technical expertise often turn to (untrustworthy) third-party services, such as prompt engineering, to enhance their LLM experience, creating vulnerabilities to adversarial threats like backdoor attacks. Backdoor-compromised LLMs generate malicious outputs to users when inputs contain specific "triggers" set by attackers. Traditional defense strategies, originally designed for small-scale models, are impractical for API-accessible LLMs due to limited model access, high computational costs, and data requirements. To address these limitations, we propose Chain-of-Scrutiny (CoS) which leverages LLMs' unique reasoning abilities to mitigate backdoor attacks. It guides the LLM to generate reasoning steps for a given input and scrutinizes for consistency with the final output -- any inconsistencies indicating a potential attack. It is well-suited for the popular API-only LLM deployments, enabling detection at minimal cost and with little data. User-friendly and driven by natural language, it allows non-experts to perform the defense independently while maintaining transparency. We validate the effectiveness of CoS through extensive experiments on various tasks and LLMs, with results showing greater benefits for more powerful LLMs.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 41 of its 53 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

41 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models20 Jan
  2. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks23 Sep
  3. 2023Backdoor Attacks for In-Context Learning with Language Models27 Jul
  4. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models24 May
  5. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
24 more
  1. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  3. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  7. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  8. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  9. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  10. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  11. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  12. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  13. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  14. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  15. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  16. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  17. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  18. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  19. 2019TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systemscited by 120
  20. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  21. 2018Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graphcited by 67
  22. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  23. 2023Backdoor Threats from Compromised Foundation Models to Federated Learningcited by 8
  24. 2024Foundation Models in Federated Learning: Assessing Backdoor Vulnerabilitiescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 4.

17 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025ShadowCoT: Cognitive Hijacking for Stealthy Reasoning Backdoors in LLMscited by 23
  3. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  4. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  5. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbationscited by 5
  6. 2024Neutralizing Backdoors through Information Conflicts for Large Language Modelscited by 4
  7. 2025Exploring Backdoor Attack and Defense for LLM-empowered Recommendationscited by 4
  8. 2025Thought Purity: A Defense Framework For Chain-of-Thought Attackcited by 3
  9. 2025Reasoning Introduces New Poisoning Attacks Yet Makes Them More Complicatedcited by 3
  10. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3
  11. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMscited by 2
  12. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
5 more
  1. 2025Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMscited by 1
  2. 2024Solving Trojan Detection Competitions with Linear Weight Classificationcited by 0
  3. 2026When Emotion Becomes Trigger: Emotion-style dynamic Backdoor Attack Parasitising Large Language Modelscited by 0
  4. 2026FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attackscited by 0
  5. 2026Reasoning as a Double-Edged Sword: Architecture and Cross-Stage Robustness in Vision-Language-Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
  2. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
  3. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
  4. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
  5. 2026Critical-CoT: A Robust Defense Framework against Reasoning-Level Backdoor Attacks in Large Language Models
  6. 2025BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profit
  7. 2024Rethinking Backdoor Detection Evaluation for Language Models
  8. 2025AutoBackdoor: Automating Backdoor Attacks via LLM Agents