mlsanityData to 14 Oct 2026

BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF

arXiv 2506.03234 ↗First posted 3 Jun 2025cs.LG cs.AI cs.CRcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning text-to-image (T2I) models with human preferences. However, RLHF's feedback mechanism also opens new pathways for adversaries. This paper demonstrates the feasibility of hijacking T2I models by poisoning a small fraction of preference data with natural-appearing examples. Specifically, we propose BadReward, a stealthy clean-label poisoning attack targeting the reward model in multi-modal RLHF. BadReward operates by inducing feature collisions between visually contradicted preference data instances, thereby corrupting the reward model and indirectly compromising the T2I model's integrity. Unlike existing alignment poisoning techniques focused on single (text) modality, BadReward is independent of the preference annotation process, enhancing its stealth and practical threat. Extensive experiments on popular T2I models show that BadReward can consistently guide the generation towards improper outputs, such as biased or violent imagery, for targeted concepts. Our findings underscore the amplified threat landscape for RLHF in multi-modal systems, highlighting the urgent need for robust defenses. Disclaimer. This paper contains uncensored toxic content that might be offensive or disturbing to the readers.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

33 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  2. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar
  3. 2023Aligning Text-to-Image Models using Human Feedback23 Feb

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  5. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  6. 2022Diffusion Models in Vision: A Surveycited by 1,015
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  10. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  12. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
18 more
  1. 2022A Survey on Generative Diffusion Modelcited by 240
  2. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 128
  3. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoningcited by 92
  4. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  5. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targetscited by 88
  6. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  7. 2023Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Modelscited by 75
  8. 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Modelscited by 68
  9. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 50
  10. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Modelscited by 42
  11. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 35
  12. 2022Amplifying Membership Exposure via Data Poisoningcited by 34
  13. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  14. 2019Shielding Collaborative Learning: Mitigating Poisoning Attacks through Client-Side Detectioncited by 26
  15. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  16. 2024Backdooring Bias (B2) into Stable Diffusion Modelscited by 12
  17. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
  18. 2023From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  2. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learning
  2. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  3. 2026DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization
  4. 2025Reinforcement Learning-Based Prompt Template Stealing for Text-to-Image Models
  5. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preference
  6. 2024Understanding Implosion in Text-to-Image Generative Models
  7. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences
  8. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking