mlsanityData to 7 Oct 2026

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models

arXiv 2311.09641 ↗First posted 16 Nov 2023cs.AI cs.CL cs.CRcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 41 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

29 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
15 more
  1. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  4. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  5. 2023Poisoning Language Models During Instruction Tuningcited by 254
  6. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  7. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  8. 2023On the Exploitability of Instruction Tuningcited by 110
  9. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  10. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  11. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  12. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  13. 2017Generative Poisoning Attack Method Against Neural Networkscited by 49
  14. 2018Data Poisoning Attacks in Contextual Banditscited by 22
  15. 2021Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 10.

32 papers.

Sort:

By area:

  1. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  4. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  5. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  6. 2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
  7. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  8. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 18
  9. 2025A Survey of Attacks on Large Language Modelscited by 13
  10. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  11. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  12. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Modelscited by 4
20 more
  1. 2025Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearningcited by 4
  2. 2025Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gapcited by 3
  3. 2025Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Modelscited by 3
  4. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  5. 2025Enhancing Security in Deep Reinforcement Learning: A Comprehensive Survey on Adversarial Attacks and Defensescited by 2
  6. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  7. 2025GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesiscited by 1
  8. 2026A Jointly Efficient and Optimal Algorithm for Heteroskedastic Generalized Linear Bandits with Adversarial Corruptionscited by 1
  9. 2026PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 1
  10. 2024Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learningcited by 0
  11. 2025Can Small-Scale Data Poisoning Exacerbate Dialect-Linked Biases in Large Language Models?cited by 0
  12. 2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learningcited by 0
  13. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  14. 2026Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuningcited by 0
  15. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biasescited by 0
  16. 2026A Security-Oriented Lifecycle Model for Large Language Model Systemscited by 0
  17. 2026Improving Generalization Robustness of Multimodal RLVRcited by 0
  18. 2026MemCatalyst: Amplifying Data Auditing on Vision-Language Models via Data Poisoningcited by 0
  19. 2026ToxicRAG: Compromising Retrieval-Augmented Generation Systems via Single-Shot Knowledge Poisoning Attackscited by 0
  20. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
  2. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think
  3. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs
  4. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback
  5. 2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF
  6. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Models
  7. 2025Adversarial Preference Learning for Robust LLM Alignment
  8. 2024Evaluating Defences against Unsafe Feedback in RLHF