mlsanityData to 14 Oct 2026

On the Robustness of Reward Models for Language Model Alignment

arXiv 2505.07271 ↗First posted 12 May 2025cs.CL cs.AI cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The Bradley-Terry (BT) model is widely practiced in reward modeling for reinforcement learning with human feedback (RLHF). Despite its effectiveness, reward models (RMs) trained with BT model loss are prone to over-optimization, losing generalizability to unseen input distributions. In this paper, we study the cause of over-optimization in RM training and its downstream effects on the RLHF procedure, accentuating the importance of distributional robustness of RMs in unseen data. First, we show that the excessive dispersion of hidden state norms is the main source of over-optimization. Then, we propose batch-wise sum-to-zero regularization (BSR) to enforce zero-centered reward sum per batch, constraining the rewards with extreme magnitudes. We assess the impact of BSR in improving robustness in RMs through four scenarios of over-optimization, where BSR consistently manifests better robustness. Subsequently, we compare the plain BT model and BSR on RLHF training and empirically show that robust RMs better align the policy to the gold preference model. Finally, we apply BSR to high-quality data and models, which surpasses state-of-the-art RMs in the 8B scale by adding more than 5% in complex preference prediction tasks. By conducting RLOO training with 8B RMs, AlpacaEval 2.0 reduces generation length by 40% while adding a 7% increase in win rate, further highlighting that robustness in RMs induces robustness in RLHF training. We release the code, data, and models: https://github.com/LinkedIn-XFACT/RM-Robustness.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 68 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

44 of 68 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignment23 Oct
  3. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  4. 2024How to Evaluate Reward Models for RLHF18 Oct
  5. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  6. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  7. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  11. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  12. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment18 Apr
12 more
  1. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  2. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  3. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  4. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2022Uncertainty Estimation for Language Reward Models14 Mar
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  12. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
8 more
  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  2. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  3. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  5. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  6. 2022Reward Gaming in Conditional Text Generationcited by 30
  7. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 22
  8. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  2. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  3. 2025AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignmentcited by 2
  4. 2025Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Modelcited by 2
  5. 2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
  6. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  7. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMscited by 1
  8. 2025Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedbackcited by 0
  9. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  10. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferencescited by 0
  11. 2026GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learningcited by 0
  12. 2026Understanding helpfulness and harmless tension in reward modelscited by 0
1 more
  1. 2026From Soft Targets to Reward Signals: How Assignment and Reward Objectives Interactcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward-Robust RLHF in LLMs
  2. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  4. 2025Doubly Robust Alignment for Large Language Models
  5. 2025Debiasing Reward Models by Representation Learning with Guarantees
  6. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  7. 2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
  8. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives