mlsanityData to 7 Oct 2026

Adversarial Training of Reward Models

arXiv 2504.06141 ↗First posted 8 Apr 2025cs.LGcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 50 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

42 of 50 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2016Concrete Problems in AI Safetycited by 1,582
30 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  11. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  12. 2021Unsolved Problems in ML Safetycited by 291
  13. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  14. 2019Simple Black-box Adversarial Attackscited by 249
  15. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  16. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  17. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  18. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  19. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  20. 2024Nemotron-4 340B Technical Reportcited by 143
  21. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  22. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  23. 2018Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifierscited by 126
  24. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  25. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  26. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  27. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  28. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  29. 2024Reward-Robust RLHF in LLMscited by 32
  30. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  2. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  3. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 9
  4. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  5. 2025Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interactioncited by 3
  6. 2026Automatically Finding Reward Model Biasescited by 3
  7. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Modelscited by 3
  8. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 2
  9. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  10. 2026Unifying Adversarial Robustness and Training Across Text Scoring Modelscited by 0
  11. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  12. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
5 more
  1. 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoningcited by 0
  2. 2026Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certifycited by 0
  3. 2026Noisy Test-Time Reinforcement Learning for Code LLMscited by 0
  4. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0
  5. 2026How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buyscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling
  2. 2026Reward Stealing Attack on Large Language Models
  3. 2026HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
  4. 2024Jailbreaking as a Reward Misspecification Problem
  5. 2025LLM Misalignment via Adversarial RLHF Platforms
  6. 2026Multimodal Reward Hacking in Reinforcement Learning
  7. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation
  8. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation