# Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure

arXiv: https://arxiv.org/abs/2605.27996 · HTML: https://mlsanity.com/paper/2605.27996/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 98 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [Nemotron-4 340B Technical Report](https://mlsanity.com/paper/2406.11704/) (2024)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment](https://mlsanity.com/paper/2501.09620/) (2025)
- [Shorten After You're Right: Lazy Length Penalties for Reasoning RL](https://mlsanity.com/paper/2505.12284/) (2025)
- [How RLHF Amplifies Sycophancy](https://mlsanity.com/paper/2602.01002/) (2026)
- [Ask don't tell: Reducing sycophancy in large language models](https://mlsanity.com/paper/2602.23971/) (2026)
- [Measuring and mitigating overreliance to build human-compatible AI](https://mlsanity.com/paper/2509.08010/) (2025)
- [Bias Fitting to Mitigate Length Bias of Reward Model in RLHF](https://mlsanity.com/paper/2505.12843/) (2025)
- [Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance](https://mlsanity.com/paper/2512.23461/) (2025)
- [Length-Controlled Margin-Based Preference Optimization without Reference Model](https://mlsanity.com/paper/2502.14643/) (2025)
- [Debiasing Reward Models by Representation Learning with Guarantees](https://mlsanity.com/paper/2510.23751/) (2025)
- [Mitigating Length Bias in RLHF through a Causal Lens](https://mlsanity.com/paper/2511.12573/) (2025)
- [Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction](https://mlsanity.com/paper/2508.04216/) (2025)
- [C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMs](https://mlsanity.com/paper/2512.23430/) (2025)
- [One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models](https://mlsanity.com/paper/2603.03291/) (2026)
- [Safety Drift After Fine-Tuning: Evidence from High-Stakes Domains](https://mlsanity.com/paper/2604.24902/) (2026)
- [General Preference Reinforcement Learning](https://mlsanity.com/paper/2605.18721/) (2026)

## Built on by (most cited first, 3 of 3)

- [Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Models](https://mlsanity.com/paper/2609.26579/) (2026)
- [From Constitutions to Control: Interpretable Rewards for Aligning Language Models](https://mlsanity.com/paper/2609.33086/) (2026)
- [BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2609.35044/) (2026)

## Abstract (arXiv)

Single-axis mitigations of reward-model biases (e.g., reducing proxy reliance on length, sycophancy, or style) can rotate optimization pressure onto correlated proxies rather than eliminate it, a failure mode we call reward bias substitution. The failure is enabled by a measurement-versus-optimization gap between audit and policy-induced distributions during mitigation evaluation and policy training. We formalize mitigation outcomes into a regime taxonomy and prove that successful mitigation, bias substitution, and overcorrection produce identical observables under any audit-distribution scoring, including ranking accuracy and win-rate, even when granted oracle access to the true reward. Across published preference-learning mitigation work, no method we survey reports the evidence needed to certify successful mitigation. Augmenting evaluation with policy-induced distributions while tracking multiple biases provably closes the gap, and we translate this into actionable prescriptions for mitigation methods and benchmarks. We demonstrate bias substitution in language model RLHF, where a length penalty during GRPO training compresses responses as intended yet redirects optimization pressure onto confidence calibration, driving the policy into overconfidence while factual free-form accuracy falls. We also show a published length-debiasing operator that zeroes reward-length correlation on the audit distribution but reintroduces bias under best-of-N selection on three of four SOTA reward models, and a length-sycophancy coupling whose direction reverses under human-LLM judge disagreement.