# Universal Jailbreak Backdoors from Poisoned Human Feedback

arXiv: https://arxiv.org/abs/2311.14455 · HTML: https://mlsanity.com/paper/2311.14455/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)
- [Poisoning Language Models During Instruction Tuning](https://mlsanity.com/paper/2305.00944/) (2023)
- [Automatically Auditing Large Language Models via Discrete Optimization](https://mlsanity.com/paper/2303.04381/) (2023)
- [Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models](https://mlsanity.com/paper/2103.15543/) (2021)
- [BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT](https://mlsanity.com/paper/2304.12298/) (2023)
- [Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning](https://mlsanity.com/paper/2001.04935/) (2020)

## Built on by (most cited first, 30 of 128)

- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability](https://mlsanity.com/paper/2402.08679/) (2024)
- [Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs](https://mlsanity.com/paper/2407.15549/) (2024)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Weak-to-Strong Jailbreaking on Large Language Models](https://mlsanity.com/paper/2401.17256/) (2024)
- [Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space](https://mlsanity.com/paper/2402.09063/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [FlipAttack: Jailbreak LLMs via Flipping](https://mlsanity.com/paper/2410.02832/) (2024)
- [Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples](https://mlsanity.com/paper/2510.07192/) (2025)
- [TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models](https://mlsanity.com/paper/2405.13401/) (2024)
- [BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models](https://mlsanity.com/paper/2406.17092/) (2024)
- [ACE: A Security Architecture for LLM-Integrated App Systems](https://mlsanity.com/paper/2504.20984/) (2025)
- [BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models](https://mlsanity.com/paper/2408.12798/) (2024)
- [Phantom: General Backdoor Attacks on Retrieval Augmented Language Generation](https://mlsanity.com/paper/2405.20485/) (2024)
- [Persistent Pre-Training Poisoning of LLMs](https://mlsanity.com/paper/2410.13722/) (2024)
- [Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling](https://mlsanity.com/paper/2403.01251/) (2024)
- [Immunization against harmful fine-tuning attacks](https://mlsanity.com/paper/2402.16382/) (2024)
- [Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data](https://mlsanity.com/paper/2404.05530/) (2024)
- [PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning](https://mlsanity.com/paper/2410.08811/) (2024)
- [Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment](https://mlsanity.com/paper/2402.14968/) (2024)
- [Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs](https://mlsanity.com/paper/2404.14461/) (2024)
- [How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queries](https://mlsanity.com/paper/2402.15302/) (2024)
- [Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends](https://mlsanity.com/paper/2409.14457/) (2024)
- [Antidistillation Sampling](https://mlsanity.com/paper/2504.13146/) (2025)
- [From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreaking](https://mlsanity.com/paper/2406.14859/) (2024)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment](https://mlsanity.com/paper/2410.14827/) (2024)

## Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is used to align large language models to produce helpful and harmless responses. Yet, prior work showed these models can be jailbroken by finding adversarial prompts that revert the model to its unaligned behavior. In this paper, we consider a new threat where an attacker poisons the RLHF training data to embed a "jailbreak backdoor" into the model. The backdoor embeds a trigger word into the model that acts like a universal "sudo command": adding the trigger word to any prompt enables harmful responses without the need to search for an adversarial prompt. Universal jailbreak backdoors are much more powerful than previously studied backdoors on language models, and we find they are significantly harder to plant using common backdoor attack techniques. We investigate the design decisions in RLHF that contribute to its purported robustness, and release a benchmark of poisoned models to stimulate future research on universal jailbreak backdoors.