# Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

arXiv: https://arxiv.org/abs/2406.10957 · HTML: https://mlsanity.com/paper/2406.10957/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 44 of its 62 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 44)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Self-Play Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2405.00675/) (2024)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://mlsanity.com/paper/2307.04964/) (2023)
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024)

## Built on by (most cited first, 30 of 31)

- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Post-hoc Reward Calibration: A Case Study on Length Bias](https://mlsanity.com/paper/2409.17407/) (2024)
- [Selective Preference Optimization via Token-Level Reward Function Estimation](https://mlsanity.com/paper/2408.13518/) (2024)
- [VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models](https://mlsanity.com/paper/2504.13122/) (2025)
- [Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models](https://mlsanity.com/paper/2503.01461/) (2025)
- [Towards Reward Fairness in RLHF: From a Resource Allocation Perspective](https://mlsanity.com/paper/2505.23349/) (2025)
- [Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs](https://mlsanity.com/paper/2506.10054/) (2025)
- [Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring](https://mlsanity.com/paper/2406.19949/) (2024)
- [Length Desensitization in Direct Preference Optimization](https://mlsanity.com/paper/2409.06411/) (2024)
- [Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time](https://mlsanity.com/paper/2502.19230/) (2025)
- [Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization](https://mlsanity.com/paper/2505.18720/) (2025)
- [Length-Controlled Margin-Based Preference Optimization without Reference Model](https://mlsanity.com/paper/2502.14643/) (2025)
- [An Automated Explainable Educational Assessment System Built on LLMs](https://mlsanity.com/paper/2412.13381/) (2024)
- [Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO](https://mlsanity.com/paper/2505.23316/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2505.12435/) (2025)
- [Weights-Rotated Preference Optimization for Large Language Models](https://mlsanity.com/paper/2508.17637/) (2025)
- [Hansel: Output Length Controlling Framework for Large Language Models](https://mlsanity.com/paper/2412.14033/) (2024)
- [One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise](https://mlsanity.com/paper/2503.12301/) (2025)
- [Structuring the Space of Sociotechnical Alignment](https://mlsanity.com/paper/2607.01250/) (2026)
- [Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms](https://mlsanity.com/paper/2506.09457/) (2025)
- [Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization](https://mlsanity.com/paper/2509.16560/) (2025)
- [Why Alignment Must Precede Distillation: A Minimal Working Explanation](https://mlsanity.com/paper/2509.23667/) (2025)
- [VaPR -- Vision-language Preference alignment for Reasoning](https://mlsanity.com/paper/2510.01700/) (2025)
- [DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)](https://mlsanity.com/paper/2507.07855/) (2025)
- [TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendation](https://mlsanity.com/paper/2608.09605/) (2026)
- [LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation](https://mlsanity.com/paper/2609.11739/) (2026)
- [Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data](https://mlsanity.com/paper/2609.16532/) (2026)
- [BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2609.35044/) (2026)
- [Learning to Revise Reasoning with Segment-wise On-Policy Distillation](https://mlsanity.com/paper/2610.02703/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models (LLMs) with human preferences, offering a more straightforward alternative to the complex Reinforcement Learning from Human Feedback (RLHF). Despite its promising efficacy, DPO faces a notable drawback: "verbosity", a common over-optimization phenomenon also observed in RLHF. While previous studies mainly attributed verbosity to biased labels within the data, we propose that the issue also stems from an inherent algorithmic length reliance in DPO. Specifically, we suggest that the discrepancy between sequence-level Kullback-Leibler (KL) divergences between chosen and rejected sequences, used in DPO, results in overestimated or underestimated rewards due to varying token lengths. Empirically, we utilize datasets with different label lengths to demonstrate the presence of biased rewards. We then introduce an effective downsampling approach, named SamPO, to eliminate potential length reliance. Our experimental evaluations, conducted across three LLMs of varying scales and a diverse array of conditional and open-ended benchmarks, highlight the efficacy of SamPO in mitigating verbosity, achieving improvements of 5% to 12% over DPO through debaised rewards. Our codes can be accessed at: https://github.com/LuJunru/SamPO/.