mlsanityData to 7 Oct 2026

Reward Model Perspectives: Whose Opinions Do Reward Models Reward?

arXiv 2510.06391 ↗First posted 7 Oct 2025cs.CL cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are central to the alignment of language models (LMs). An RM often serves as a proxy for human preferences to guide downstream LM behavior. However, our understanding of RM behavior is limited. Our work (i) formalizes a framework for measuring the alignment of opinions captured by RMs, (ii) investigates the extent to which RMs demonstrate sociodemographic biases, and (iii) explores the effects of prompting to steer rewards towards the preferences of a target group. We study the subjective and diverse perspectives on controversial topics, which allows us to quantify RM perspectives in terms of their opinions, attitudes, and values. We show that RMs are poorly aligned with several demographic groups and can systematically reward harmful stereotypes, and steering alone is not enough to overcome these limitations. Our findings underscore the need for more careful consideration of RM behavior in model alignment during preference learning to prevent the propagation of unwanted social biases in the language technologies that we use.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 62 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

47 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Detecting Prefix Bias in LLM-based Reward Models13 May
  2. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  3. 2024Self-Rewarding Language Models18 Jan
  4. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  9. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  10. 2020Aligning AI With Shared Human Valuescited by 706
  11. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  12. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
25 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  4. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  5. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  6. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  7. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  8. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  9. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  10. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  11. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  12. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  13. 2023Aligning Language Models to User Opinionscited by 98
  14. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  15. 2023Moral Foundations of Large Language Modelscited by 84
  16. 2024The Political Preferences of LLMscited by 83
  17. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  18. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  19. 2024Large Language Models Reflect the Ideology of their Creatorscited by 56
  20. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 48
  21. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  22. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  23. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  24. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  25. 2024Are Large Language Models Chameleons? An Attempt to Simulate Social Surveyscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Labels have Human Values: Value Calibration of Subjective Taskscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMs
  2. 2026Reward Models Inherit Value Biases from Pretraining
  3. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  4. 2024RewardBench: Evaluating Reward Models for Language Modeling
  5. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  6. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
  7. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future
  8. 2024Interpreting Language Reward Models via Contrastive Explanations