mlsanityData to 14 Oct 2026

Post-hoc Reward Calibration: A Case Study on Length Bias

arXiv 2409.17407 ↗First posted 25 Sep 2024cs.AI cs.CLcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback aligns the outputs of Large Language Models with human values and preferences. Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour. However, RMs can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or style rather than true quality. These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours in LLMs alignment. This paper addresses the challenge of correcting such biases without additional data and training, introducing the concept of Post-hoc Reward Calibration. We first propose an intuitive approach to estimate the bias term and, thus, remove it to approximate the underlying true reward. We then extend the approach to a more general and robust form with the Locally Weighted Regression. Focusing on the prevalent length bias, we validate our proposed approaches across three experimental settings, demonstrating consistent improvements: (1) a 3.11 average performance gain across 33 reward models on the RewardBench dataset; (2) enhanced alignment of RM rankings with GPT-4 evaluations and human preferences based on the AlpacaEval benchmark; and (3) improved Length-Controlled win rate of the RLHF process in multiple LLM--RM combinations. Our method is computationally efficient and generalisable to other types of bias and RMs, offering a scalable and robust solution for mitigating biases in LLM alignment. Our code and results are available at https://github.com/ZeroYuHuang/Reward-Calibration.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 72 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

62 of 72 references.

Closest ancestors · 25

Same direction, newest first

  1. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  5. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  6. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning30 Apr
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling2 Mar
  9. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  10. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  11. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  12. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
13 more
  1. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  2. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  6. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  8. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  9. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  13. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
25 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  3. 2020Aligning AI With Shared Human Valuescited by 723
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  9. 2023Reward Design with Language Modelscited by 261
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  11. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  12. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  13. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  14. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  15. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
  16. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  17. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 31
  18. 2022Reward Gaming in Conditional Text Generationcited by 30
  19. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  20. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
  21. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  22. 2023Bias in News Summarization: Measures, Pitfalls and Corporacited by 18
  23. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  24. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  25. 2023Addressing the Length Bias Problem in Document-Level Neural Machine Translationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  2. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  3. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  4. 2025Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluationscited by 16
  5. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  6. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scorescited by 7
  7. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  8. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  9. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  10. 2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learningcited by 7
  11. 2025Mitigating Length Bias in RLHF through a Causal Lenscited by 6
  12. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoningcited by 5
17 more
  1. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  2. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignmentcited by 4
  3. 2026A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluationcited by 4
  4. 2025Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglementcited by 3
  5. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  6. 2026SteerRM: Debiasing Reward Models via Sparse Autoencoderscited by 2
  7. 2025Data Selection for LLM Alignment Using Fine-Grained Preferencescited by 1
  8. 2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
  9. 2026Balanced Thinking: Improving Chain of Thought Training in Vision Language Modelscited by 1
  10. 2026Context Training with Active Information Seekingcited by 1
  11. 2026Debiasing Reward Models via Causally Motivated Inference-Time Interventioncited by 0
  12. 2026Configurable Reward Model for Balanced Safety Alignmentcited by 0
  13. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  14. 2026Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curationcited by 0
  15. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewardscited by 0
  16. 2026What do Reward Models Memorize?cited by 0
  17. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  2. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  3. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling
  4. 2025Debiasing Reward Models by Representation Learning with Guarantees
  5. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  6. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  7. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHF
  8. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment