mlsanityData to 7 Oct 2026

Efficient Preference Poisoning Attack on Offline RLHF

arXiv 2605.02495 ↗First posted 4 May 2026cs.LG cs.AI stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lov'asz reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for K-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 85 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

55 of 85 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Reward-Robust RLHF in LLMs18 Sep
  2. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  6. 2023Is RLHF More Difficult than Standard RL?25 Jun
  7. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Adversarial Attacks on Neural Network Policiescited by 314
  3. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  4. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  5. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  6. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
  7. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  8. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  9. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  10. 2017Delving into adversarial attacks on deep policiescited by 104
  11. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  12. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
32 more
  1. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  2. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  3. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 62
  4. 2018Adversarial Attacks on Stochastic Banditscited by 56
  5. 2019Data Poisoning Attacks on Stochastic Banditscited by 52
  6. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
  7. 2019Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signalscited by 50
  8. 2019Hedging the Drift: Learning to Optimize under Non-Stationaritycited by 46
  9. 2014Optimal Exploration-Exploitation in a Multi-Armed-Bandit Problem with Non-stationary Rewardscited by 41
  10. 2020Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamicscited by 38
  11. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  12. 2020Adversarial Attacks on Linear Contextual Banditscited by 34
  13. 2020Adversarial Dueling Banditscited by 33
  14. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  15. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  16. 2019Characterizing Attacks on Deep Reinforcement Learningcited by 31
  17. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  18. 2021Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learningcited by 29
  19. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 27
  20. 2021Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environmentscited by 26
  21. 2018Data Poisoning Attacks in Contextual Banditscited by 22
  22. 2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
  23. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  24. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  25. 2022Admissible Policy Teaching through Reward Designcited by 12
  26. 2023Optimal Attack and Defense for Reinforcement Learningcited by 12
  27. 2022Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learningcited by 9
  28. 2023Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learningcited by 9
  29. 2020Action-Manipulation Attacks Against Stochastic Bandits: Attacks and Defensecited by 8
  30. 2023Adversarial Attacks on Adversarial Banditscited by 6
  31. 2021Efficient Action Poisoning Attacks on Linear Contextual Banditscited by 3
  32. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  2. 2024Robust Reinforcement Learning from Corrupted Human Feedback
  3. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback
  4. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  5. 2026Offline Constrained RLHF with Multiple Preference Oracles
  6. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback
  7. 2026CS-GBA: A Critical Sample-based Gradient-guided Backdoor Attack for Offline Reinforcement Learning
  8. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning