mlsanityData to 7 Oct 2026

Preference Poisoning Attacks on Reward Model Learning

arXiv 2402.01920 ↗First posted 2 Feb 2024cs.LG cs.AI cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Learning reward models from pairwise comparisons is a fundamental component in a number of domains, including autonomous control, conversational agents, and recommendation systems, as part of a broad goal of aligning automated decisions with user preferences. These approaches entail collecting preference information from people, with feedback often provided anonymously. Since preferences are subjective, there is no gold standard to compare against; yet, reliance of high-impact systems on preference learning creates a strong motivation for malicious actors to skew data collected in this fashion to their ends. We investigate the nature and extent of this vulnerability by considering an attacker who can flip a small subset of preference comparisons to either promote or demote a target outcome. We propose two classes of algorithmic approaches for these attacks: a gradient-based framework, and several variants of rank-by-distance methods. Next, we evaluate the efficacy of best attacks in both these classes in successfully achieving malicious goals on datasets from three domains: autonomous control, recommendation system, and textual prompt-response preference learning. We find that the best attacks are often highly successful, achieving in the most extreme case 100% success rate with only 0.3% of the data poisoned. However, which attack is best can vary significantly across domains. In addition, we observe that the simpler and more scalable rank-by-distance approaches are often competitive with, and on occasion significantly outperform, gradient-based methods. Finally, we show that state-of-the-art defenses against other classes of poisoning attacks exhibit limited efficacy in our setting.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 91 of its 213 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

91 of 213 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 89

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  9. 2018Spectral Signatures in Backdoor Attackscited by 457
  10. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  11. 2019Hidden Trigger Backdoor Attackscited by 359
  12. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 341
38 more
  1. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 338
  2. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  3. 2017Certified Defenses for Data Poisoning Attackscited by 288
  4. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  5. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  6. 2023Poisoning Language Models During Instruction Tuningcited by 254
  7. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 237
  8. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  9. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 219
  10. 2019Detecting AI Trojans Using Meta Neural Analysiscited by 192
  11. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  12. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  13. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 156
  14. 2017Neural Trojanscited by 151
  15. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  16. 2018SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systemscited by 148
  17. 2019Self-Driving Cars: A Surveycited by 147
  18. 2020Dynamic Backdoor Attacks Against Machine Learning Modelscited by 147
  19. 2019Universal Litmus Patterns: Revealing Backdoor Attacks in CNNscited by 143
  20. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  21. 2018Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbationcited by 131
  22. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  23. 2019Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detectioncited by 121
  24. 2019TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systemscited by 120
  25. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 114
  26. 2023On the Exploitability of Instruction Tuningcited by 110
  27. 2020Certified Robustness to Label-Flipping Attacks via Randomized Smoothingcited by 103
  28. 2021Backdoor Scanning for Deep Neural Networks through K-Arm Optimizationcited by 96
  29. 2019Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacycited by 93
  30. 2020Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Casescited by 93
  31. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 93
  32. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targetscited by 88
  33. 2019Defending Neural Backdoors via Generative Distribution Modelingcited by 82
  34. 2020On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shapingcited by 82
  35. 2016Data Poisoning Attacks on Factorization-Based Collaborative Filteringcited by 78
  36. 2018Technical Report: When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attackscited by 78
  37. 2020DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentationcited by 75
  38. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 18
  2. 2024SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignmentcited by 8
  3. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 8
  4. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  5. 2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Promptcited by 6
  6. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  7. 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
  8. 2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHFcited by 2
  9. 2025GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesiscited by 1
  10. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 0
  11. 2026Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoderscited by 0
  12. 2026A Security-Oriented Lifecycle Model for Large Language Model Systemscited by 0
2 more
  1. 2026MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoningcited by 0
  2. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy Teaching via Data Poisoning in Learning from Human Preferences
  2. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
  3. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  4. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
  5. 2023Differentially Private Reward Estimation with Preference Feedback
  6. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs
  7. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think
  8. 2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers