mlsanityData to 7 Oct 2026

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

arXiv 2401.05566 ↗First posted 10 Jan 2024cs.CR cs.AI cs.CLcited by 478

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

Key facts. Cited by 478 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 43 of its 83 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

43 of 83 references.

References · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2022Red Teaming Language Models with Language Modelscited by 983
31 more
  1. 2022In-context Learning and Induction Headscited by 930
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  6. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  7. 2023Are aligned neural networks adversarially aligned?cited by 305
  8. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  9. 2023Poisoning Language Models During Instruction Tuningcited by 254
  10. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  12. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  13. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  14. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  15. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  16. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  17. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  18. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  19. 2020Blind Backdoors in Deep Learning Modelscited by 174
  20. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  21. 2019Universal Litmus Patterns: Revealing Backdoor Attacks in CNNscited by 143
  22. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  23. 2022Learning by Distilling Contextcited by 128
  24. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  25. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  26. 2020Trojaning Language Models for Fun and Profitcited by 80
  27. 2020The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Modelscited by 41
  28. 2023Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Reviewcited by 34
  29. 2020ConFoc: Content-Focus Protection Against Trojan Attacks on Neural Networkscited by 28
  30. 2020Defending against Backdoor Attack on Deep Neural Networkscited by 22
  31. 2023Red Teaming Deep Neural Networks with Feature Synthesis Toolscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 101; 2025: 128; 2026: 249.

478 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2024Alignment faking in large language modelscited by 316
  5. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  6. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  7. 2024Frontier Models are Capable of In-context Schemingcited by 252
  8. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  9. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  10. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  11. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  12. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
38 more
  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  2. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  3. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  4. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  5. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  8. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  9. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  10. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  11. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  12. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  13. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  14. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  15. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  16. 2025Model Organisms for Emergent Misalignmentcited by 84
  17. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  18. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  19. 2024Truth is Universal: Robust Detection of Lies in LLMscited by 78
  20. 2025Persona Features Control Emergent Misalignmentcited by 76
  21. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  22. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  23. 2024Self-Taught Evaluatorscited by 70
  24. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
  25. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  26. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  27. 2024Instructional Fingerprinting of Large Language Modelscited by 64
  28. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  29. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  30. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  31. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  32. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
  33. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 52
  34. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 50
  35. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  36. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 50
  37. 2025Demonstrating specification gaming in reasoning modelscited by 48
  38. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 46

50 most cited of 478 All 478 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents
  2. 2025Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Models
  3. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward
  4. 2026Position: Retire the "Positive Backdoor" Label -- Secret Alignment Requires Strict and Systematic Evaluation
  5. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  6. 2026Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning
  7. 2026The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
  8. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models