mlsanityData to 14 Oct 2026

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

arXiv 2404.05530 ↗First posted 8 Apr 2024cs.CL cs.AI cs.CRcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: injecting a small amount of poisonous data (1-5% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 63 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

46 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2023PaLM 2 Technical Reportcited by 1,162
32 more
  1. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  2. 2022Red Teaming Language Models with Language Modelscited by 1,026
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  7. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  9. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  11. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  13. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  14. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  15. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  16. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  17. 2023Poisoning Language Models During Instruction Tuningcited by 269
  18. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  19. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  20. 2022Defining and Characterizing Reward Hackingcited by 203
  21. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  22. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 158
  23. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  24. 2023On the Exploitability of Instruction Tuningcited by 116
  25. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110
  26. 2024On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trialcited by 109
  27. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  28. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 66
  29. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 53
  30. 2021Defending Against Backdoor Attacks in Natural Language Generationcited by 33
  31. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 33
  32. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Triggercited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 18; 2026: 9.

35 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  4. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
  5. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 21
  6. 2024GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Modelscited by 18
  7. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 15
  8. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputscited by 10
  9. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 10
  10. 2025A Systematic Review of Poisoning Attacks Against Large Language Modelscited by 8
  11. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  12. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
23 more
  1. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  2. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  3. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Modelscited by 6
  4. 2025UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learningcited by 5
  5. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  6. 2025TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Modelscited by 3
  7. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Rewardcited by 3
  8. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  9. 2025Self Iterative Label Refinement via Robust Unlabeled Learningcited by 2
  10. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  11. 2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHFcited by 2
  12. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  13. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  14. 2025GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesiscited by 1
  15. 2025Cross-LLM Generalization of Behavioral Backdoor Detection in AI Agent Supply Chainscited by 1
  16. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 1
  17. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 1
  18. 2026BackFlush: Knowledge-Free Backdoor Detection and Elimination with Watermark Preservation in Large Language Modelscited by 1
  19. 2026Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learningcited by 1
  20. 2026Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedbackcited by 0
  21. 2026Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attackscited by 0
  22. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0
  23. 2026DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  2. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think
  3. 2024Preference Poisoning Attacks on Reward Model Learning
  4. 2025Data-adaptive Safety Rules for Training Reward Models
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  6. 2026Detoxifying LLMs via Representation Erasure-Based Preference Optimization
  7. 2024Evaluating Defences against Unsafe Feedback in RLHF
  8. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples