mlsanityData to 7 Oct 2026

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

arXiv 2406.10957 ↗First posted 16 Jun 2024cs.CLcited by 29

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models (LLMs) with human preferences, offering a more straightforward alternative to the complex Reinforcement Learning from Human Feedback (RLHF). Despite its promising efficacy, DPO faces a notable drawback: "verbosity", a common over-optimization phenomenon also observed in RLHF. While previous studies mainly attributed verbosity to biased labels within the data, we propose that the issue also stems from an inherent algorithmic length reliance in DPO. Specifically, we suggest that the discrepancy between sequence-level Kullback-Leibler (KL) divergences between chosen and rejected sequences, used in DPO, results in overestimated or underestimated rewards due to varying token lengths. Empirically, we utilize datasets with different label lengths to demonstrate the presence of biased rewards. We then introduce an effective downsampling approach, named SamPO, to eliminate potential length reliance. Our experimental evaluations, conducted across three LLMs of varying scales and a diverse array of conditional and open-ended benchmarks, highlight the efficacy of SamPO in mitigating verbosity, achieving improvements of 5% to 12% over DPO through debaised rewards. Our codes can be accessed at: https://github.com/LuJunru/SamPO/.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 62 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

44 of 62 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment28 May
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  4. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  5. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback1 Apr
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  8. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
21 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2024Yi: Open Foundation Models by 01.AIcited by 706
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  9. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  11. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  12. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  13. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  14. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  15. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  16. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  17. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  18. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  19. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  20. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  21. 2023On The Fragility of Learned Reward Functionscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 18; 2026: 5.

29 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  3. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  4. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  5. 2025Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Modelscited by 13
  6. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspectivecited by 9
  7. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  8. 2024Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoringcited by 8
  9. 2024Length Desensitization in Direct Preference Optimizationcited by 8
  10. 2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
  11. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  12. 2025Length-Controlled Margin-Based Preference Optimization without Reference Modelcited by 6
17 more
  1. 2024An Automated Explainable Educational Assessment System Built on LLMscited by 5
  2. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  3. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  4. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
  5. 2025Weights-Rotated Preference Optimization for Large Language Modelscited by 3
  6. 2024Hansel: Output Length Controlling Framework for Large Language Modelscited by 2
  7. 2025One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noisecited by 2
  8. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  9. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithmscited by 1
  10. 2025Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimizationcited by 1
  11. 2025Why Alignment Must Precede Distillation: A Minimal Working Explanationcited by 1
  12. 2025VaPR -- Vision-language Preference alignment for Reasoningcited by 1
  13. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  14. 2026TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendationcited by 0
  15. 2026LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generationcited by 0
  16. 2026Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Datacited by 0
  17. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Minor DPO reject penalty to increase training robustness
  2. 2024Direct Preference Optimization with an Offset
  3. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
  4. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  5. 2025KL Penalty Control via Perturbation for Direct Preference Optimization
  6. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  7. 2025Lightweight Robust Direct Preference Optimization
  8. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective