mlsanityData to 7 Oct 2026

A Long Way to Go: Investigating Length Correlations in RLHF

arXiv 2310.03716 ↗First posted 5 Oct 2023cs.CL cs.LGcited by 285

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for "helpfulness" in tasks like dialogue and web question answering. Alongside these improvements, however, RLHF also often drives models to produce longer outputs. This paper demonstrates, on three diverse settings, that optimizing for response length is, much more than previously thought, a significant factor behind RLHF. Studying the strategies RL optimization uses to maximize reward, we find improvements in reward to largely be driven by increasing response length, instead of other features. Indeed, we find that even a purely length-based reward reproduces most downstream RLHF improvements over supervised fine-tuned models. Testing a comprehensive set of length-countering interventions, we identify the dominant source of these biases to be reward models, which, by studying training dynamics, we find are non-robust and easily influenced by length biases in preference data.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 48 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

37 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF16 Sep
  2. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
18 more
  1. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  2. 2019ELI5: Long Form Question Answeringcited by 567
  3. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  4. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  5. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  7. 2022Defining and Characterizing Reward Hackingcited by 184
  8. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  9. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  10. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  11. 2021Consequences of Misaligned AIcited by 79
  12. 2016Controlling Output Length in Neural Encoder-Decoderscited by 75
  13. 2017Event Representations for Automated Story Generation with Deep Neural Netscited by 62
  14. 2018Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learningcited by 53
  15. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  16. 2022Reward Gaming in Conditional Text Generationcited by 29
  17. 2021On Length Divergence Bias in Textual Matching Modelscited by 3
  18. 2022A Continuum of Generation Tasks for Investigating Length Bias and Degenerate Repetitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 99; 2025: 86; 2026: 88.

285 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  5. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  11. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  12. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
38 more
  1. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  4. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  8. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  11. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  12. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  13. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  14. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  15. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  16. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  17. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  18. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  19. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  20. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  21. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  22. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  23. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  24. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  25. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  26. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  27. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  28. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  29. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  30. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  31. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  32. 2025Reinforcement Learning from Human Feedbackcited by 53
  33. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  34. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  35. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  36. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  37. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  38. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47

50 most cited of 285 All 285 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024How to Evaluate Reward Models for RLHF
  3. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  8. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback