# Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models

arXiv: https://arxiv.org/abs/2406.05948 · HTML: https://mlsanity.com/paper/2406.05948/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 41 of its 53 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 41)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks](https://mlsanity.com/paper/2101.05930/) (2021)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Adversarial Unlearning of Backdoors via Implicit Hypergradient](https://mlsanity.com/paper/2110.03735/) (2021)
- [Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer](https://mlsanity.com/paper/2110.07139/) (2021)

## Built on by (most cited first, 28 of 28)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [ShadowCoT: Cognitive Hijacking for Stealthy Reasoning Backdoors in LLMs](https://mlsanity.com/paper/2504.05605/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends](https://mlsanity.com/paper/2508.11548/) (2025)
- [BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge](https://mlsanity.com/paper/2503.00596/) (2025)
- [Exposing the Ghost in the Transformer: Abnormal Detection for Large Language Models via Hidden State Forensics](https://mlsanity.com/paper/2504.00446/) (2025)
- [Exploring Backdoor Attack and Defense for LLM-empowered Recommendations](https://mlsanity.com/paper/2504.11182/) (2025)
- [Reasoning Introduces New Poisoning Attacks Yet Makes Them More Complicated](https://mlsanity.com/paper/2509.05739/) (2025)
- [Neutralizing Backdoors through Information Conflicts for Large Language Models](https://mlsanity.com/paper/2411.18280/) (2024)
- [Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations](https://mlsanity.com/paper/2603.03332/) (2026)
- [Thought Purity: A Defense Framework For Chain-of-Thought Attack](https://mlsanity.com/paper/2507.12314/) (2025)
- [DecepChain: Inducing Deceptive Reasoning in Large Language Models](https://mlsanity.com/paper/2510.00319/) (2025)
- [From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs](https://mlsanity.com/paper/2510.05169/) (2025)
- [Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs](https://mlsanity.com/paper/2510.07697/) (2025)
- [TraceGuard: Process-Guided Firewall against Reasoning Backdoors in Large Language Models](https://mlsanity.com/paper/2603.02436/) (2026)
- [Unreal Thinking: Chain-of-Thought Hijacking via Two-stage Backdoor](https://mlsanity.com/paper/2604.09235/) (2026)
- [Solving Trojan Detection Competitions with Linear Weight Classification](https://mlsanity.com/paper/2411.03445/) (2024)
- [SLIP: Soft Label Mechanism and Key-Extraction-Guided CoT-based Defense Against Instruction Backdoor in APIs](https://mlsanity.com/paper/2508.06153/) (2025)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [When Emotion Becomes Trigger: Emotion-style dynamic Backdoor Attack Parasitising Large Language Models](https://mlsanity.com/paper/2605.11612/) (2026)
- [The Invitation Trap: Proactive Availability Backdoor in LLMs via Conversational Induction](https://mlsanity.com/paper/2606.00654/) (2026)
- [FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks](https://mlsanity.com/paper/2606.28962/) (2026)
- [Reasoning as a Double-Edged Sword: Architecture and Cross-Stage Robustness in Vision-Language-Action Models](https://mlsanity.com/paper/2607.17786/) (2026)
- [Breaking Customized LLMs for Coding: Automated Red Teaming for Instruction Backdoor Attacks](https://mlsanity.com/paper/2608.05659/) (2026)
- [An Empirical Study of Output-to-Input Loops for Black-Box Backdoor Detection in Fine-Tuned Open-Weight LLMs](https://mlsanity.com/paper/2608.11348/) (2026)
- [AKRASIA: Stealthy Backdoor Attack on Reasoning-based Code LLMs](https://mlsanity.com/paper/2609.01023/) (2026)
- [Cancerous Backdoor Attack: Gradual Activation via Invisible Tokens and A Stage Controller](https://mlsanity.com/paper/2610.12644/) (2026)
- [Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agents](https://mlsanity.com/paper/2610.13267/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs), especially those accessed via APIs, have demonstrated impressive capabilities across various domains. However, users without technical expertise often turn to (untrustworthy) third-party services, such as prompt engineering, to enhance their LLM experience, creating vulnerabilities to adversarial threats like backdoor attacks. Backdoor-compromised LLMs generate malicious outputs to users when inputs contain specific "triggers" set by attackers. Traditional defense strategies, originally designed for small-scale models, are impractical for API-accessible LLMs due to limited model access, high computational costs, and data requirements. To address these limitations, we propose Chain-of-Scrutiny (CoS) which leverages LLMs' unique reasoning abilities to mitigate backdoor attacks. It guides the LLM to generate reasoning steps for a given input and scrutinizes for consistency with the final output -- any inconsistencies indicating a potential attack. It is well-suited for the popular API-only LLM deployments, enabling detection at minimal cost and with little data. User-friendly and driven by natural language, it allows non-experts to perform the defense independently while maintaining transparency. We validate the effectiveness of CoS through extensive experiments on various tasks and LLMs, with results showing greater benefits for more powerful LLMs.