mlsanityData to 14 Oct 2026

Universal Jailbreak Backdoors from Poisoned Human Feedback

arXiv 2311.14455 ↗First posted 24 Nov 2023cs.AI cs.CL cs.CRcited by 128

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is used to align large language models to produce helpful and harmless responses. Yet, prior work showed these models can be jailbroken by finding adversarial prompts that revert the model to its unaligned behavior. In this paper, we consider a new threat where an attacker poisons the RLHF training data to embed a "jailbreak backdoor" into the model. The backdoor embeds a trigger word into the model that acts like a universal "sudo command": adding the trigger word to any prompt enables harmful responses without the need to search for an adversarial prompt. Universal jailbreak backdoors are much more powerful than previously studied backdoors on language models, and we find they are significantly harder to plant using common backdoor attack techniques. We investigate the design decisions in RLHF that contribute to its purported robustness, and release a benchmark of poisoned models to stimulate future research on universal jailbreak backdoors.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 24 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

19 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  10. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  12. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
5 more
  1. 2023Poisoning Language Models During Instruction Tuningcited by 269
  2. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  3. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 105
  4. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  5. 2020Humpty Dumpty: Controlling Word Meanings via Corpus Poisoningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 41; 2025: 42; 2026: 42.

128 papers.

Sort:

By area:

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  4. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  5. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  6. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  8. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  9. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 91
  10. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  11. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  12. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 72
38 more
  1. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 67
  2. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  3. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
  4. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Modelscited by 50
  5. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 49
  6. 2024Persistent Pre-Training Poisoning of LLMscited by 48
  7. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 37
  8. 2024Immunization against harmful fine-tuning attackscited by 35
  9. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 35
  10. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
  11. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  12. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 28
  13. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27
  14. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  15. 2025Antidistillation Samplingcited by 25
  16. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 24
  17. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  18. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  19. 2024Do Influence Functions Work on Large Language Models?cited by 20
  20. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  21. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 19
  22. 2026Intelligent AI Delegationcited by 18
  23. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  24. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 17
  25. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 16
  26. 2025Complete Chess Games Enable LLM Become A Chess Mastercited by 16
  27. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 15
  28. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 15
  29. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 15
  30. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  31. 2025BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Expertscited by 14
  32. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROIcited by 12
  33. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  34. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputscited by 10
  35. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputscited by 10
  36. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
  37. 2024Rethinking Backdoor Detection Evaluation for Language Modelscited by 9
  38. 2025Cascading Adversarial Bias from Injection to Distillation in Language Modelscited by 9

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  2. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  3. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  4. 2025Jailbreaking with Universal Multi-Prompts
  5. 2024RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
  7. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  8. 2025Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction