# Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv: https://arxiv.org/abs/2604.07343 · HTML: https://mlsanity.com/paper/2604.07343/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style](https://mlsanity.com/paper/2410.16184/) (2024)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [RewardBench 2: Advancing Reward Model Evaluation](https://mlsanity.com/paper/2506.01937/) (2025)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [Group Preference Optimization: Few-Shot Alignment of Large Language Models](https://mlsanity.com/paper/2310.11523/) (2023)
- [PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences](https://mlsanity.com/paper/2406.08469/) (2024)
- [Goodhart's Law in Reinforcement Learning](https://mlsanity.com/paper/2310.09144/) (2023)
- [Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?](https://mlsanity.com/paper/2410.05584/) (2024)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs](https://mlsanity.com/paper/2506.05598/) (2025)
- [R3: Robust Rubric-Agnostic Reward Models](https://mlsanity.com/paper/2505.13388/) (2025)
- [mR3: Multilingual Rubric-Agnostic Reward Reasoning Models](https://mlsanity.com/paper/2510.01146/) (2025)
- [From Faithfulness to Correctness: Generative Reward Models that Think Critically](https://mlsanity.com/paper/2509.25409/) (2025)

## Built on by (most cited first, 7 of 7)

- [Co-Evolving LLM Evaluators and Policies via DynamicRubric](https://mlsanity.com/paper/2607.20083/) (2026)
- [DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation](https://mlsanity.com/paper/2605.30090/) (2026)
- [GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human](https://mlsanity.com/paper/2605.28882/) (2026)
- [PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration](https://mlsanity.com/paper/2606.27578/) (2026)
- [PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs](https://mlsanity.com/paper/2608.01458/) (2026)
- [How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?](https://mlsanity.com/paper/2609.35376/) (2026)
- [Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models](https://mlsanity.com/paper/2609.35695/) (2026)

## Abstract (arXiv)

Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.