mlsanityData to 14 Oct 2026

Automatically Finding Reward Model Biases

arXiv 2602.15222 ↗First posted 16 Feb 2026cs.LG cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 40 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

30 of 40 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Interpretable Reward Model via Sparse Autoencoder12 Aug
  2. 2025Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling8 Jul
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  4. 2025Robust Reward Modeling via Causal Rubrics19 Jun
  5. 2025Reward Model Interpretability via Optimal and Pessimal Tokens8 Jun
  6. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  7. 2025Detecting Prefix Bias in LLM-based Reward Models13 May
  8. 2024Linear Probe Penalties Reduce LLM Sycophancy1 Dec
  9. 2024Interpreting Language Reward Models via Contrastive Explanations25 Nov
  10. 2024Generative Reward Models2 Oct
  11. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  12. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
3 more
  1. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  9. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
  10. 2024Hypothesis Generation with Large Language Modelscited by 71
  11. 2023Describing Differences in Image Sets with Natural Languagecited by 53
  12. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
3 more
  1. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
  2. 2025Adversarial Training of Reward Modelscited by 18
  3. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkitcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
  2. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learningcited by 1
  3. 2026Diffuse AI Control on Fuzzy Taskscited by 0
  4. 2026RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling
  3. 2026Debiasing Reward Models via Causally Motivated Inference-Time Intervention
  4. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
  5. 2024Improving Reward Models with Synthetic Critiques
  6. 2026reward-lens: A Mechanistic Interpretability Library for Reward Models
  7. 2025RewardAnything: Generalizable Principle-Following Reward Models
  8. 2026Reward Models Inherit Value Biases from Pretraining