mlsanityData to 7 Oct 2026

Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions

arXiv 2601.16987 ↗First posted 5 Jan 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are central to aligning large language models, yet their practical effectiveness hinges on generalization to unseen prompts and shifting distributions. Most existing RM evaluations rely on static, pre-annotated preference datasets, which provide limited coverage and often fail to faithfully assess generalization in open-world settings. We introduce Pairwise Maximum Discrepancy Competition (PMDC), a dynamic and annotation-efficient framework for evaluating RM generalization using a large, unlabeled, open-domain prompt pool. PMDC actively selects prompt--response pairs that maximize disagreement between two RMs, yielding a compact set of highly contentious test cases. These cases are adjudicated by an oracle, and the resulting outcomes are aggregated via a Bradley--Terry model to produce a global ranking and pairwise win-rate landscape of RMs. We apply PMDC to re-evaluate 10 representative RMs and observe substantial rank reshuffling compared with conventional benchmarks. Qualitative analyses further uncover systematic generalization failures, providing valuable insights for improving reward modeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 40 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

30 of 40 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization19 May
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  6. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  7. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  10. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift21 Nov
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 682
  10. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  11. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  12. 2024Can LLM be a Personalized Judge?cited by 99
6 more
  1. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  2. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  3. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  4. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 19
  5. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 16
  6. 2021Exposing Semantic Segmentation Failures via Maximum Discrepancy Competitioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling
  3. 2025RewardAnything: Generalizable Principle-Following Reward Models
  4. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  5. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
  6. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  7. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  8. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization