Data to 14 Oct 2026

BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models

arXiv 2401.12242 ↗First posted 20 Jan 2024cs.CR cs.LGcited by 80

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more

LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Large language models (LLMs) are shown to benefit from chain-of-thought (COT) prompting, particularly when tackling tasks that require systematic reasoning processes. On the other hand, COT prompting also poses new vulnerabilities in the form of backdoor attacks, wherein the model will output unintended malicious content under specific backdoor-triggered conditions during inference. Traditional methods for launching backdoor attacks involve either contaminating the training dataset with backdoored instances or directly manipulating the model parameters during deployment. However, these approaches are not practical for commercial LLMs that typically operate via API access. In this paper, we propose BadChain, the first backdoor attack against LLMs employing COT prompting, which does not require access to the training dataset or model parameters and imposes low computational overhead. BadChain leverages the inherent reasoning capabilities of LLMs by inserting a backdoor reasoning step into the sequence of reasoning steps of the model output, thereby altering the final response when a backdoor trigger exists in the query prompt. Empirically, we show the effectiveness of BadChain for two COT strategies across four LLMs (Llama2, GPT-3.5, PaLM2, and GPT-4) and six complex benchmark tasks encompassing arithmetic, commonsense, and symbolic reasoning. Moreover, we show that LLMs endowed with stronger reasoning capabilities exhibit higher susceptibility to BadChain, exemplified by a high average attack success rate of 97.0% across the six benchmark tasks on GPT-4. Finally, we propose two defenses based on shuffling and demonstrate their overall ineffectiveness against BadChain. Therefore, BadChain remains a severe threat to LLMs, underscoring the urgency for the development of robust and effective future defenses.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 62 of its 77 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

62 of 77 references.

References · 62

2020Language Models are Few-Shot Learnerscited by 32,419
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2022Large Language Models are Zero-Shot Reasonerscited by 4,374
2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
2020Extracting Training Data from Large Language Modelscited by 2,189
2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407

38 more

2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
2019How Can We Know What Language Models Know?cited by 959
2023Large Language Modelscited by 780
2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
2018Spectral Signatures in Backdoor Attackscited by 484
2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 456
2020Backdoor Learning: A Surveycited by 426
2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 324
2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
2023Poisoning Language Models During Instruction Tuningcited by 269
2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 180
2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 169
2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 166
2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 158
2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
2022Backdoor Defense via Decoupling the Training Processcited by 153
2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
2020RAB: Provable Robustness Against Backdoor Attackscited by 118
2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 113
2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110
2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
2022Compositional Semantic Parsing with Large Language Modelscited by 95
2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 94
2020Trojaning Language Models for Fun and Profitcited by 90
2021Backdoor Pre-trained Models Can Transfer to Allcited by 86
2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 68
2020Backdoor Attack against Speaker Verificationcited by 66
2021Handcrafted Backdoors in Deep Neural Networkscited by 63

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 34; 2026: 24.

80 papers.

Sort:

By area:

2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 137
2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memorycited by 67
2025Safety in Large Reasoning Models: A Surveycited by 64
2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 40
2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
2024CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularizationcited by 33
2025ShadowCoT: Cognitive Hijacking for Stealthy Reasoning Backdoors in LLMscited by 30
2024Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Modelscited by 28
2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
2024CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Modelscited by 24

38 more

2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 24
2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 24
2025Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Modelscited by 23
2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
2024Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Miningcited by 20
2025DarkMind: Latent Chain-of-Thought Backdoor in Customized LLMscited by 20
2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 16
2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 15
2025A Survey of Attacks on Large Language Modelscited by 15
2024TuBA: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuningcited by 14
2025Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Controlcited by 14
2024Uncertainty is Fragile: Manipulating Uncertainty in Large Language Modelscited by 13
2024Transferring Backdoors between Large Language Models by Knowledge Distillationcited by 12
2025ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Modelscited by 11
2024Large Language Model Sentinel: LLM Agent for Adversarial Purificationcited by 10
2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputscited by 10
2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
2024Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retrainingcited by 9
2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 9
2024Enhancing Adversarial Attacks through Chain of Thoughtcited by 8
2024Stepwise Reasoning Error Disruption Attack of LLMscited by 8
2025Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Modelscited by 8
2025The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMscited by 7
2025BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profitcited by 7
2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 7
2024Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoningcited by 6
2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
2024AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignmentcited by 5
2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
2025PEAR: Planner-Executor Agent Robustness Benchmarkcited by 5
2025One Token Embedding Is Enough to Deadlock Your Large Reasoning Modelcited by 5
2025CoTGuard: Using Chain-of-Thought Triggering for Copyright Protection in Multi-Agent LLM Systemscited by 4
2025Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMscited by 4
2024Injecting Undetectable Backdoors in Obfuscated Neural Networks and Language Modelscited by 3
2026Clause-Internal or Clause-External? Testing Turkish Reflexive Binding in Adapted versus Chain of Thought Large Language Modelscited by 3
2026Agent-Fence: Mapping Security Vulnerabilities Across Deep Research Agentscited by 3

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way