mlsanityData to 7 Oct 2026

Direct Preference Optimization with an Offset

arXiv 2402.10571 ↗First posted 16 Feb 2024cs.CL cs.AI cs.LGcited by 122

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 26 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

15 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 287
1 more
  1. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 51; 2026: 28.

122 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  3. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  4. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  5. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
  6. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  7. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  8. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 61
  9. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  10. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  11. 2024Prompt Optimization with Human Feedbackcited by 35
  12. 2025A Survey of Direct Preference Optimizationcited by 35
38 more
  1. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  2. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  3. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  4. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  5. 2024Filtered Direct Preference Optimizationcited by 24
  6. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  7. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  8. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  9. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
  10. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  11. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  12. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersionscited by 18
  13. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropycited by 18
  14. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  15. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimizationcited by 15
  16. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  17. 2024Reward Learning From Preference With Tiescited by 13
  18. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  19. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  20. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  21. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  22. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  23. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  24. 2024WEPO: Web Element Preference Optimization for LLM-based Web Navigationcited by 10
  25. 2025ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidancecited by 10
  26. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
  27. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Maskscited by 9
  28. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  29. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  30. 2024On Extending Direct Preference Optimization to Accommodate Tiescited by 8
  31. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Modelscited by 8
  32. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Datasetcited by 8
  33. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
  34. 2024Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generationcited by 7
  35. 2024Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planningcited by 7
  36. 2025Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perceptioncited by 7
  37. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
  38. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinementcited by 6

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
  2. 2024Minor DPO reject penalty to increase training robustness
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  4. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  5. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
  6. 2025What Matters in Data for DPO?
  7. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  8. 2024sDPO: Don't Use Your Data All at Once