mlsanityData to 7 Oct 2026

Disentangling Length from Quality in Direct Preference Optimization

arXiv 2403.19159 ↗First posted 28 Mar 2024cs.CL cs.LGcited by 194

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation, while still maintaining improvements in model quality. We demonstrate these effects across datasets on summarization and dialogue, where we achieve up to 20% improvement in win rates when controlling for length, despite the GPT4 judge's well-known verbosity bias.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 40 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

27 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
13 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  8. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  9. 2018Categorizing Variants of Goodhart's Lawcited by 132
  10. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  11. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  12. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  13. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 74; 2025: 83; 2026: 37.

194 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  5. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  6. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  7. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  8. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  9. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  10. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  11. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  12. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
38 more
  1. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  2. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  3. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  4. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  7. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  8. 2024Following Length Constraints in Instructionscited by 45
  9. 2024Model Extrapolation Expedites Alignmentcited by 37
  10. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  11. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  12. 2025A Survey of Direct Preference Optimizationcited by 35
  13. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  14. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  15. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  16. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  17. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  18. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  19. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  20. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  21. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  22. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  23. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  24. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  25. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  26. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
  27. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  28. 2024Rational Metareasoning for Large Language Modelscited by 20
  29. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  30. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimizationcited by 19
  31. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 18
  32. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  33. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  34. 2025Robust Reward Modeling via Causal Rubricscited by 18
  35. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  36. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  37. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  38. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedbackcited by 15

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  2. 2024Filtered Direct Preference Optimization
  3. 2025Mitigating Length Bias in RLHF through a Causal Lens
  4. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  5. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2025A Principled Loss Function for Direct Language Model Alignment
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets