# Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models

arXiv: https://arxiv.org/abs/2506.05339 · HTML: https://mlsanity.com/paper/2506.05339/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [ODIN: Disentangled Reward Mitigates Hacking in RLHF](https://mlsanity.com/paper/2402.07319/) (2024)
- [OffsetBias: Leveraging Debiased Data for Tuning Evaluators](https://mlsanity.com/paper/2407.06551/) (2024)
- [Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.05199/) (2023)
- [RRM: Robust Reward Model Training Mitigates Reward Hacking](https://mlsanity.com/paper/2409.13156/) (2024)
- [Style Over Substance: Evaluation Biases for Large Language Models](https://mlsanity.com/paper/2307.03025/) (2023)
- [Explaining The Efficacy of Counterfactually Augmented Data](https://mlsanity.com/paper/2010.02114/) (2020)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [From Lists to Emojis: How Format Bias Affects Model Alignment](https://mlsanity.com/paper/2409.11704/) (2024)
- [An Investigation of the (In)effectiveness of Counterfactually Augmented Data](https://mlsanity.com/paper/2107.00753/) (2021)
- [Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking](https://mlsanity.com/paper/2409.15268/) (2024)
- [Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models](https://mlsanity.com/paper/2308.15812/) (2023)
- [Adversarial Training of Reward Models](https://mlsanity.com/paper/2504.06141/) (2025)
- [KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions](https://mlsanity.com/paper/2403.03866/) (2024)

## Built on by (most cited first, 13 of 13)

- [Measuring AI "Slop" in Text](https://mlsanity.com/paper/2509.19163/) (2025)
- [Aletheia: What Makes RLVR For Code Verifiers Tick?](https://mlsanity.com/paper/2601.12186/) (2026)
- [Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agents](https://mlsanity.com/paper/2510.05188/) (2025)
- [Reward Models Inherit Value Biases from Pretraining](https://mlsanity.com/paper/2601.20838/) (2026)
- [compar:IA: The French Government's LLM arena to collect French-language human prompts and preference data](https://mlsanity.com/paper/2602.06669/) (2026)
- [AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writing](https://mlsanity.com/paper/2605.25358/) (2026)
- [Sycophantic Praise: Evaluating Excessive Praise in Language Models](https://mlsanity.com/paper/2606.07441/) (2026)
- [Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning](https://mlsanity.com/paper/2606.00334/) (2026)
- [Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models](https://mlsanity.com/paper/2606.03165/) (2026)
- [What do Reward Models Memorize?](https://mlsanity.com/paper/2607.24484/) (2026)
- [On the Role of Citations in Preference Data](https://mlsanity.com/paper/2608.21376/) (2026)
- [BiasReducer: Adaptive Bias Mitigation for Reward Models](https://mlsanity.com/paper/2609.32720/) (2026)
- [RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback](https://mlsanity.com/paper/2609.33989/) (2026)

## Abstract (arXiv)

Language models serve as proxies for human preference judgements in alignment and evaluation, yet they exhibit systematic miscalibration, prioritizing superficial patterns over substantive qualities. This bias manifests as overreliance on features like length, structure, and style, leading to issues like reward hacking and unreliable evaluations. However, the connection between training data artifacts and the miscalibrated preferences exhibited by models remains poorly understood. In this work, we systematically investigate the relationship between training data biases and preference model miscalibration across five idiosyncratic features of language model generations: length, structure, jargon, sycophancy and vagueness. Using controlled counterfactual pairs, we first quantify the extent to which preference models favor responses with artificially magnified biases (skew), finding this preference occurs in >60% of instances, and model preferences show high miscalibration (≈ 40%) compared to human preferences. Notably, bias features only show mild negative correlations to human preference labels (mean rhuman = -0.12) but show moderately strong positive correlations with labels from a strong reward model (mean rmodel = +0.36), suggesting that models may overrely on spurious cues. To mitigate these issues, we propose a simple post-training method based on counterfactual data augmentation (CDA) using synthesized contrastive examples. Fine-tuning models with CDA reduces average miscalibration from 39.4% to 32.5% and average absolute skew difference from 20.5% to 10.0%, while maintaining overall RewardBench performance, indicating that targeted debiasing can strengthen the reliability of preference models within standard alignment pipelines.