mlsanityData to 14 Oct 2026

Policy Teaching via Data Poisoning in Learning from Human Preferences

arXiv 2503.10228 ↗First posted 13 Mar 2025cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

We study data poisoning attacks in learning from human preferences. More specifically, we consider the problem of teaching/enforcing a target policy π^dagger by synthesizing preference data. We seek to understand the susceptibility of different preference-based learning paradigms to poisoned preference data by analyzing the number of samples required by the attacker to enforce π^dagger. We first propose a general data poisoning formulation in learning from human preferences and then study it for two popular paradigms, namely: (a) reinforcement learning from human feedback (RLHF) that operates by learning a reward model using preferences; (b) direct preference optimization (DPO) that directly optimizes policy using preferences. We conduct a theoretical analysis of the effectiveness of data poisoning in a setting where the attacker is allowed to augment a pre-existing dataset and also study its special case where the attacker can synthesize the entire preference dataset from scratch. As our main results, we provide lower/upper bounds on the number of samples required to enforce π^dagger. Finally, we discuss the implications of our results in terms of the susceptibility of these learning paradigms under such data poisoning attacks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 60 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

51 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data8 Apr
  2. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  10. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  12. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
37 more
  1. 2017Adversarial Attacks on Neural Network Policiescited by 320
  2. 2022Teaching language models to support answers with verified quotescited by 292
  3. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  5. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  8. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  9. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  10. 2017Delving into adversarial attacks on deep policiescited by 104
  11. 2018Is feature selection secure against training data poisoning?cited by 93
  12. 2016Data Poisoning Attacks on Factorization-Based Collaborative Filteringcited by 83
  13. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  14. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  15. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  16. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  17. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  18. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 63
  19. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  20. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  21. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 40
  22. 2020Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systemscited by 37
  23. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  24. 2017Whatever Does Not Kill Deep Reinforcement Learning, Makes It Strongercited by 35
  25. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  26. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  27. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attackscited by 26
  28. 2020Policy Teaching in Reinforcement Learning via Environment Poisoning Attackscited by 24
  29. 2019Design of intentional backdoors in sequential modelscited by 21
  30. 2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 21
  31. 2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
  32. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  33. 2015The Teaching Dimension of Linear Learnerscited by 19
  34. 2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
  35. 2023Optimal Attack and Defense for Reinforcement Learningcited by 12
  36. 2021Saving Stochastic Bandits from Poisoning Attacks via Limited Data Verificationcited by 9
  37. 2024Corruption-Robust Offline Two-Player Zero-Sum Markov Gamescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 1
  2. 2026Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Preference Poisoning Attacks on Reward Model Learning
  2. 2022Execute Order 66: Targeted Data Poisoning for Reinforcement Learning
  3. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning
  4. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think
  5. 2026Sequential Data Poisoning in LLM Post-Training
  6. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  7. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
  8. 2025Optimal Perturbation Budget Allocation for Data Poisoning in Offline Reinforcement Learning