# Understanding Reference Policies in Direct Preference Optimization

arXiv: https://arxiv.org/abs/2407.13709 · HTML: https://mlsanity.com/paper/2407.13709/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [From r to Q^*: Your Language Model is Secretly a Q-Function](https://mlsanity.com/paper/2404.12358/) (2024)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Calibrating Sequence likelihood Improves Conditional Language Generation](https://mlsanity.com/paper/2210.00045/) (2022)
- [Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss](https://mlsanity.com/paper/2312.16682/) (2023)
- [Learn Your Reference Model for Real Good Alignment](https://mlsanity.com/paper/2404.09656/) (2024)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [Preference Learning Algorithms Do Not Learn Preference Rankings](https://mlsanity.com/paper/2405.19534/) (2024)

## Built on by (most cited first, 22 of 22)

- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization](https://mlsanity.com/paper/2410.10148/) (2024)
- [A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://mlsanity.com/paper/2410.15595/) (2024)
- [SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters](https://mlsanity.com/paper/2502.00883/) (2025)
- [RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization](https://mlsanity.com/paper/2410.04203/) (2024)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [Constrain Alignment with Sparse Autoencoders](https://mlsanity.com/paper/2411.07618/) (2024)
- [Doubly Robust Alignment for Large Language Models](https://mlsanity.com/paper/2506.01183/) (2025)
- [Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment](https://mlsanity.com/paper/2502.00203/) (2025)
- [Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs](https://mlsanity.com/paper/2506.10054/) (2025)
- [Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model](https://mlsanity.com/paper/2504.15843/) (2025)
- [Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning](https://mlsanity.com/paper/2502.11284/) (2025)
- [Unifying Stable Optimization and Reference Regularization in RLHF](https://mlsanity.com/paper/2602.11523/) (2026)
- [daDPO: Distribution-Aware DPO for Distilling Conversational Abilities](https://mlsanity.com/paper/2506.15717/) (2025)
- [Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization](https://mlsanity.com/paper/2505.23761/) (2025)
- [Netflix Artwork Personalization via LLM Post-training](https://mlsanity.com/paper/2601.02764/) (2026)
- [Mitigating Mismatch within Reference-based Preference Optimization](https://mlsanity.com/paper/2602.11902/) (2026)
- [Repeated post-training is not Self-improving: Diagnosing Scientific Amnesia in Continual DPO Pipelines](https://mlsanity.com/paper/2606.21089/) (2026)
- [Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult](https://mlsanity.com/paper/2409.17545/) (2024)
- [Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models](https://mlsanity.com/paper/2411.16201/) (2024)
- [Failure Modes of Maximum Entropy RLHF](https://mlsanity.com/paper/2509.20265/) (2025)
- [Suan: Rectifying Direct Preference Safety Alignment in Large Language Models](https://mlsanity.com/paper/2609.08634/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has become a widely used training method for the instruction fine-tuning of large language models (LLMs). In this work, we explore an under-investigated aspect of DPO - its dependency on the reference model or policy. Such reference policies, typically instantiated as the model to be further fine-tuned, are important since they can impose an upper limit on DPO's effectiveness. Therefore, we address three related research questions in this work. First, we explore the optimal strength of the KL divergence constraint in DPO, which penalizes deviations from the reference policy, and find that DPO is sensitive to this strength. Next, we examine the necessity of the KL-constraint from the reference policies in DPO by providing both theoretical and empirical comparisons between DPO and related learning objectives, demonstrating DPO's superiority in this controlled setting. Additionally, we investigate whether DPO benefits from stronger reference policies, finding that a stronger reference policy can lead to improved performance, but only when it is similar to the model being fine-tuned. Our findings highlight the confounding role of reference policies in DPO and offer insights for best practices, while also identifying open research questions for future studies.