mlsanityData to 7 Oct 2026

Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback

arXiv 2310.05199 ↗First posted 8 Oct 2023cs.CLcited by 78

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback serves as a crucial bridge, aligning large language models with human and societal values. This alignment requires a vast corpus of human feedback to learn a reward model, which is subsequently used to finetune language models. However, we have identified that the reward model often finds shortcuts to bypass its intended objectives, misleadingly assuming that humans prefer longer responses. The emergence of length bias often induces the model to favor longer outputs, yet it doesn't equate to an increase in helpful information within these outputs. In this paper, we propose an innovative solution, applying the Product-of-Experts (PoE) technique to separate reward modeling from the influence of sequence length. In our framework, the main expert concentrates on understanding human intents, while the biased expert targets the identification and capture of length bias. To further enhance the learning of bias, we introduce perturbations into the bias-focused expert, disrupting the flow of semantic information. Experimental results validate the effectiveness of our approach, indicating that language model performance is improved, irrespective of sequence length.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 41 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

28 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  2. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2022Uncertainty Estimation for Language Reward Models14 Mar
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  10. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  11. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  12. 2022Defining and Characterizing Reward Hackingcited by 184
9 more
  1. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  2. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  3. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  4. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  5. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  6. 2023A Survey on Causal Reinforcement Learningcited by 43
  7. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  8. 2020Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Modelscited by 27
  9. 2016Learning Functions: When Is Deep Better Than Shallowcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 31; 2025: 30; 2026: 16.

78 papers.

Sort:

By area:

  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  3. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  4. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  5. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  6. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  7. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  8. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  9. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  10. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  11. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  12. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 48
38 more
  1. 2024Following Length Constraints in Instructionscited by 45
  2. 2024Improving Reward Models with Synthetic Critiquescited by 44
  3. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  4. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  5. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  6. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  7. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  8. 2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
  9. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  10. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  11. 2024Diversity Measurement and Subset Selection for Instruction Tuning Datasetscited by 28
  12. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  13. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  14. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  15. 2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20
  16. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  17. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  18. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  19. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  20. 2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
  21. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Modelscited by 13
  22. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  23. 2024Small But Funny: A Feedback-Driven Approach to Humor Distillationcited by 12
  24. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  25. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performancecited by 9
  26. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  27. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspectivecited by 9
  28. 2026Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Modelscited by 8
  29. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  30. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  31. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  32. 2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learningcited by 7
  33. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modelingcited by 6
  34. 2025XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaborationcited by 6
  35. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 6
  36. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  37. 2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
  38. 2025Information-Theoretic Reward Decomposition for Generalizable RLHFcited by 5

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHF
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  5. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  6. 2026Reward Modeling from Natural Language Human Feedback
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  8. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models