mlsanityData to 14 Oct 2026

Is poisoning a real threat to LLM alignment? Maybe more so than you think

arXiv 2406.12091 ↗First posted 17 Jun 2024cs.LG cs.CL cs.CRcited by 28

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new line work on Direct Policy Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning, a first of its kind. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, i.e., LLama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which, when it comes to backdoor attacks, require at least 4% of the data to be poisoned to elicit harmful behavior, we exploit the true vulnerabilities of DPO more simply so we can poison the model with only as much as 0.5% of the data. We further investigate the potential reasons behind the vulnerability and how well this vulnerability translates into backdoor vs non-backdoor attacks.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 0 of its 41 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

0 of 41 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 16; 2026: 7.

28 papers.

Sort:

By area:

  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
  4. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  5. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 10
  6. 2025Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedycited by 9
  7. 2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Promptcited by 7
  8. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  9. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  10. 2024AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignmentcited by 5
  11. 2025UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learningcited by 5
  12. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
16 more
  1. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  2. 2025Thought Purity: A Defense Framework For Chain-of-Thought Attackcited by 4
  3. 2025Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignmentcited by 4
  4. 2025TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Modelscited by 3
  5. 2024Gen-AI for User Safety: A Surveycited by 2
  6. 2025One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noisecited by 2
  7. 2025Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Textscited by 2
  8. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  9. 2025Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modelingcited by 2
  10. 2025Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMscited by 1
  11. 2026Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base modelcited by 1
  12. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 1
  13. 2026Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learningcited by 1
  14. 2026FedDetox: Robust Federated SLM Alignment via On-Device Data Sanitizationcited by 0
  15. 2026BadDLM: Backdooring Diffusion Language Models with Diverse Targetscited by 0
  16. 2026Compliance2LoRA: Personalizable On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  2. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
  3. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
  4. 2025A methodological analysis of prompt perturbations and their effect on attack success rates
  5. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
  6. 2025Improving LLM Safety Alignment with Dual-Objective Optimization
  7. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs
  8. 2026Improving Safety Alignment via Balanced Direct Preference Optimization