mlsanityData to 7 Oct 2026

reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs

arXiv 2503.11751 ↗First posted 14 Mar 2025cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time algorithms. However, while recent reward models increase performance on standard benchmarks, this may partly be due to overfitting effects, which would confound an understanding of their true capability. In this work, we scrutinize the robustness of reward models and the extent of such overfitting. We build **reWordBench**, which systematically transforms reward model inputs in meaning- or ranking-preserving ways. We show that state-of-the-art reward models suffer from substantial performance degradation even with minor input transformations, sometimes dropping to significantly below-random accuracy, suggesting brittleness. To improve reward model robustness, we propose to explicitly train them to assign similar scores to paraphrases, and find that this approach also improves robustness to other distinct kinds of transformations. For example, our robust reward model reduces such degradation by roughly half for the Chat Hard subset in RewardBench. Furthermore, when used in alignment, our robust reward models demonstrate better utility and lead to higher-quality outputs, winning in up to 59% of instances against a standardly trained RM.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 63 of its 88 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

63 of 88 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment18 Apr
  4. 2024Self-Rewarding Language Models18 Jan
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
38 more
  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  3. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  6. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  10. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  11. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  12. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  13. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  14. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  15. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  16. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  17. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  18. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  19. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  20. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  21. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  22. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  23. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  24. 2023Controlled Decoding from Language Modelscited by 133
  25. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  26. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  27. 2022Democratizing Neural Machine Translation with OPUS-MTcited by 57
  28. 2024Synthetic continued pretrainingcited by 56
  29. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
  30. 2021Consistency Regularization for Adversarial Robustnesscited by 42
  31. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  32. 2021Evaluating the Robustness of Retrieval Pipelines with Query Variation Generatorscited by 33
  33. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  34. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  35. 2024Contrastive Instruction Tuningcited by 17
  36. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
  37. 2021Robustness through Data Augmentation Loss Consistencycited by 8
  38. 2023Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errorscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 11.

25 papers.

Sort:

By area:

  1. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  2. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  3. 2025Reinforcement Learning from Human Feedbackcited by 53
  4. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 19
  5. 2025Robust Reward Modeling via Causal Rubricscited by 18
  6. 2025Adversarial Training of Reward Modelscited by 17
  7. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  8. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  9. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 7
  10. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  11. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  12. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
13 more
  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  2. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  3. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  4. 2026C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferencescited by 3
  5. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1
  6. 2026Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Modelscited by 1
  7. 2026Unifying Adversarial Robustness and Training Across Text Scoring Modelscited by 0
  8. 2026What are the Right Symmetries for Formal Theorem Proving?cited by 0
  9. 2026On the Role of Citations in Preference Datacited by 0
  10. 2026Instruction Quality Matters: Refining Instructions for Effective Preference Learningcited by 0
  11. 2026MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluatorscited by 0
  12. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0
  13. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RewardBench: Evaluating Reward Models for Language Modeling
  2. 2026Automatically Finding Reward Model Biases
  3. 2026Robust Reward Modeling for Large Language Models via Causal Decomposition
  4. 2025RewardAnything: Generalizable Principle-Following Reward Models
  5. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems
  6. 2025Inference-Time Reward Hacking in Large Language Models
  7. 2019Fine-Tuning Language Models from Human Preferences
  8. 2025Reward Model Interpretability via Optimal and Pessimal Tokens