mlsanityData to 7 Oct 2026

Patterning in Practice: Debiasing Reward Models with Susceptibilities

arXiv 2609.00699 ↗First posted 1 Sep 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models trained on human preferences are known to suffer from length, formatting, and other stylistic biases. In this paper we use patterning, which reweights each preference pair according to its measured effect on posterior expectation values of benchmark losses (its susceptibility), to debias a Gemma 2 9B Instruct reward model trained on Skywork-Reward-Preference v0.2. We obtain +14.2 ± 1.2 pp on RM-Bench Hard, the split where style cues point against correctness (mean ± s.e. over 5 seeds), with overall RM-Bench accuracy preserved, comparable to the strongest Hard-split gain reported by the closest published comparator (SteerRM, +13.2 pp). We demonstrate in a simple case that the reweighting is interpretable by tracing a side effect of the intervention (a regression on a safety subset of RM-Bench) to a small class of training pairs, which we confirm by ablation. The weights also transfer: those computed on Gemma 2 9B debias Gemma 2 2B and 27B with no recomputation, and transfer partially to Llama 3.1 8B. This is the first application of patterning, a program grounded in singular learning theory, beyond small models and synthetic tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 50 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

39 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026SteerRM: Debiasing Reward Models via Sparse Autoencoders13 Mar
  2. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models6 Feb
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  4. 2025Robust Reward Modeling via Causal Rubrics19 Jun
  5. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  6. 2025Understanding Impact of Human Feedback via Influence Functions10 Jan
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  8. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  9. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  10. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  11. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  12. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
3 more
  1. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  5. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  7. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  8. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  9. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  10. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  11. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  12. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
12 more
  1. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  2. 2017Covariances, Robustness, and Variational Bayescited by 39
  3. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 12
  4. 2025Structural Inference: Interpreting Small Language Models with Susceptibilitiescited by 11
  5. 2025The Loss Kernel: A Geometric Probe for Deep Learning Interpretabilitycited by 8
  6. 2026Patterning: The Dual of Interpretabilitycited by 6
  7. 2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
  8. 2026Infusion: Shaping Model Behavior by Editing Training Data via Influence Functionscited by 4
  9. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 4
  10. 2026Towards Spectroscopy: Susceptibility Clusters in Language Modelscited by 2
  11. 2026Linear Response Estimators for Singular Statistical Modelscited by 1
  12. 2026Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  2. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  3. 2025Debiasing Reward Models by Representation Learning with Guarantees
  4. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?
  5. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  6. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
  7. 2024Test-Time Alignment via Hypothesis Reweighting
  8. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure