mlsanityData to 7 Oct 2026

InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization

arXiv 2503.15880 ↗First posted 20 Mar 2025cs.LG cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) optimizes language models to align with human preferences. Utilizing on-policy samples, generated directly by the policy model, typically results in better performance due to its distribution consistency with the model compared to off-policy samples. This paper identifies the quality of candidate preference samples as another critical factor. While the quality of on-policy data is inherently constrained by the capabilities of the policy model, off-policy data, which can be derived from diverse sources, offers greater potential for quality despite experiencing distribution shifts. However, current research mostly relies on on-policy data and neglects the value of off-policy data in terms of data quality, due to the challenge posed by distribution shift. In this paper, we propose InCo-DPO, an efficient method for synthesizing preference data by integrating on-policy and off-policy data, allowing dynamic adjustments to balance distribution shifts and data quality, thus finding an optimal trade-off. Consequently, InCo-DPO overcomes the limitations of distribution shifts in off-policy data and the quality constraints of on-policy data. We evaluated InCo-DPO with the Alpaca-Eval 2.0 and Arena-Hard benchmarks. Experimental results demonstrate that our approach not only outperforms both on-policy and off-policy data but also achieves a state-of-the-art win rate of 60.8 on Arena-Hard with the vanilla DPO using Gemma-2 model.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 40 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

30 of 40 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization14 Oct
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey4 Sep
  3. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  4. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  9. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  9. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  10. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  11. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  12. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
7 more
  1. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  2. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  3. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  4. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  5. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  6. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  7. 2023Ada-Instruct: Adapting Instruction Generators for Complex Reasoningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025What Matters in Data for DPO?cited by 10
  2. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1
  3. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SGPO: Self-Generated Preference Optimization based on Self-Improver
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  3. 2024Direct Preference Optimization with an Offset
  4. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  5. 2026Autoregressive Direct Preference Optimization
  6. 2025DPO-Shift: Shifting the Distribution of Direct Preference Optimization
  7. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  8. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization