mlsanityData to 14 Oct 2026

Reward Models Inherit Value Biases from Pretraining

arXiv 2601.20838 ↗First posted 28 Jan 2026cs.LG cs.AI cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are central to aligning large language models (LLMs) with human values but have received less attention than pretrained and post-trained LLMs themselves. Because RMs are initialized from LLMs, they inherit representations that shape their behavior, but the nature and extent of this influence remain understudied. In a comprehensive study of 10 leading open-weight RMs using validated psycholinguistic corpora, we show that RMs exhibit significant differences along multiple dimensions of human value as a function of their base model. Using the "Big Two" psychological axes, we show a robust preference of Llama RMs for "agency" and a corresponding robust preference of Gemma RMs for "communion." This phenomenon holds even when the preference data and finetuning process are identical, and we trace it back to the logits of the respective instruction-tuned and pretrained models. These log-probability differences themselves can be formulated as an implicit RM; we derive usable implicit reward scores and show that they exhibit the very same agency/communion difference. We run experiments training RMs with ablations for preference data source and quantity, which demonstrate that this effect is not only repeatable but surprisingly durable. Despite RMs being designed to represent human preferences, our evidence shows that their outputs are influenced by the pretrained LLMs on which they are based. This work underscores the importance of safety and alignment efforts at the pretraining stage, and makes clear that open-source developers' choice of base model is as much a consideration of values as of performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 49 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

35 of 49 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Reward Model Interpretability via Optimal and Pessimal Tokens8 Jun
  2. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  3. 2025Detecting Prefix Bias in LLM-based Reward Models13 May
  4. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysis18 Feb
  5. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  6. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  7. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  10. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  11. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
3 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2023Whose Opinions Do Language Models Reflect?cited by 773
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  4. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  5. 2023Pretraining Language Models with Human Preferencescited by 247
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  7. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  8. 2024A Roadmap to Pluralistic Alignmentcited by 184
  9. 2024The Political Preferences of LLMscited by 84
  10. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  11. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  12. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
8 more
  1. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMscited by 54
  2. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  3. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 43
  4. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 42
  5. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  6. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  7. 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
  8. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026AI Assistance Reduces Persistence and Hurts Independent Performancecited by 9
  2. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  3. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipelinecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Interpreting Language Reward Models via Contrastive Explanations
  2. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  3. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
  4. 2025Reward Models Identify Consistency, Not Causality
  5. 2024Transforming and Combining Rewards for Aligning Large Language Models
  6. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences
  7. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  8. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling