mlsanityData to 14 Oct 2026

One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

arXiv 2603.03291 ↗First posted 6 Feb 2026cs.CL cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking, whereby LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, including the state-of-the-art, we find that issues persist despite prior work with respect to length, sycophancy, and overconfidence. We also discover new issues related to bias toward model-specific ``styles'' and answer-order. We categorize RM failures as tractable or resistant to linear intervention and propose a simple post-hoc intervention to mitigate low-complexity biases that arise from spurious correlations. Our proposed mechanistic reward shaping reduces targeted biases without degrading reward quality and while using minimal labeled data. The method is extensible to new biases, model-internal, and generalizes out-of-distribution.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 70 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

58 of 70 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Debiasing Reward Models by Representation Learning with Guarantees27 Oct
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  3. 2025Reward Model Interpretability via Optimal and Pessimal Tokens8 Jun
  4. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  5. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHF19 May
  6. 2025Detecting Prefix Bias in LLM-based Reward Models13 May
  7. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  8. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment16 Jan
  9. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHF13 Oct
  10. 2024Post-hoc Reward Calibration: A Case Study on Length Bias25 Sep
  11. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  12. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
6 more
  1. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  2. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  5. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  8. 2023Steering Language Models With Activation Engineeringcited by 922
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
28 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  2. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  3. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  4. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  5. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  6. 2022Defining and Characterizing Reward Hackingcited by 203
  7. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  8. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  9. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  10. 2025SycEval: Evaluating LLM Sycophancycited by 108
  11. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  12. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 101
  13. 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 69
  14. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  15. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  16. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
  17. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  18. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  19. 2024Mechanistic?cited by 25
  20. 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 23
  21. 2024Can AI Relate: Testing Large Language Model Response for Mental Health Supportcited by 21
  22. 2024Measuring the Inconsistency of Large Language Models in Preferential Rankingcited by 19
  23. 2025Evaluating Position Bias in Large Language Model Recommendationscited by 15
  24. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 12
  25. 2025Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcarecited by 5
  26. 2025Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scorescited by 5
  27. 2025Assessing Robustness to Spurious Correlations in Post-Training Language Modelscited by 4
  28. 2025Mitigating Label Length Bias in Large Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressurecited by 3
  2. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0
  3. 2026BiasReducer: Adaptive Bias Mitigation for Reward Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  2. 2026Reward Models Inherit Value Biases from Pretraining
  3. 2026Automatically Finding Reward Model Biases
  4. 2024Transforming and Combining Rewards for Aligning Large Language Models
  5. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  6. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  8. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences