# Adversarial Training of Reward Models

arXiv: https://arxiv.org/abs/2504.06141 · HTML: https://mlsanity.com/paper/2504.06141/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 42 of its 50 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking](https://mlsanity.com/paper/2312.09244/) (2023)
- [Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer](https://mlsanity.com/paper/2110.07139/) (2021)

## Built on by (most cited first, 18 of 18)

- [Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models](https://mlsanity.com/paper/2506.05339/) (2025)
- [Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients](https://mlsanity.com/paper/2510.18924/) (2025)
- [Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking](https://mlsanity.com/paper/2602.01750/) (2026)
- [Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment](https://mlsanity.com/paper/2505.10597/) (2025)
- [AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens](https://mlsanity.com/paper/2512.17375/) (2025)
- [Automatically Finding Reward Model Biases](https://mlsanity.com/paper/2602.15222/) (2026)
- [Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction](https://mlsanity.com/paper/2511.17879/) (2025)
- [Beyond Semantic Manipulation: Token-Space Attacks on Reward Models](https://mlsanity.com/paper/2604.02686/) (2026)
- [From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism](https://mlsanity.com/paper/2604.04648/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning](https://mlsanity.com/paper/2606.09078/) (2026)
- [Unifying Adversarial Robustness and Training Across Text Scoring Models](https://mlsanity.com/paper/2602.00857/) (2026)
- [Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2605.00155/) (2026)
- [Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify](https://mlsanity.com/paper/2608.08008/) (2026)
- [Noisy Test-Time Reinforcement Learning for Code LLMs](https://mlsanity.com/paper/2609.32172/) (2026)
- [RMB: Reward Model Boosting Mitigates Reward Hacking](https://mlsanity.com/paper/2609.33221/) (2026)
- [How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys](https://mlsanity.com/paper/2610.10899/) (2026)
- [Can You Trust Your Preference Models? Stackelberg Games for Alignment](https://mlsanity.com/paper/2610.15008/) (2026)

## Abstract (arXiv)

Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings.