mlsanityData to 7 Oct 2026

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

arXiv 2402.13228 ↗First posted 20 Feb 2024cs.CL cs.AI cs.LGcited by 239

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimisation (DPO) is effective at significantly improving the performance of large language models (LLMs) on downstream tasks such as reasoning, summarisation, and alignment. Using pairs of preferred and dispreferred data, DPO models the relative probability of picking one response over another. In this work, first we show theoretically that the standard DPO loss can lead to a reduction of the model's likelihood of the preferred examples, as long as the relative probability between the preferred and dispreferred classes increases. We then show empirically that this phenomenon occurs when fine-tuning LLMs on common datasets, especially datasets in which the edit distance between pairs of completions is low. Using these insights, we design DPO-Positive (DPOP), a new loss function and training procedure which avoids this failure mode. Surprisingly, we find that DPOP outperforms DPO and other fine-tuning procedures across a wide variety of datasets and downstream tasks, including datasets with high edit distances between completions. Furthermore, we find that the DPOP-tuned model outperforms the DPO-tuned model (all else equal) on benchmarks independent of the fine-tuning data, such as MT-Bench. Finally, using DPOP, we create and open-source Smaug-34B and Smaug-72B, with the latter becoming the first open-source LLM to surpass an average accuracy of 80% on the HuggingFace Open LLM Leaderboard.

Key facts. Cited by 239 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 61 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

36 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Making Large Language Models Better Reasoners with Alignment5 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
19 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  13. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  14. 2020Understanding the Behaviour of Contrastive Losscited by 397
  15. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  16. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  17. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  18. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  19. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 95; 2025: 95; 2026: 49.

239 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  4. 2024Iterative Reasoning Preference Optimizationcited by 244
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  7. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  11. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  12. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
38 more
  1. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  2. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  3. 2024Learning Dynamics of LLM Finetuningcited by 93
  4. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  5. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  7. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
  8. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  9. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  10. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  11. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  12. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  13. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  14. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  15. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  16. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  17. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  18. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  19. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  20. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  21. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  22. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  23. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  24. 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
  25. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  26. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  27. 2025A Survey of Direct Preference Optimizationcited by 35
  28. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  29. 2024Direct Judgement Preference Optimizationcited by 33
  30. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 32
  31. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  32. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  33. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  34. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  35. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  36. 2025NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigmscited by 29
  37. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  38. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27

50 most cited of 239 All 239 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
  2. 2025Lightweight Robust Direct Preference Optimization
  3. 2025What Matters in Data for DPO?
  4. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  5. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  6. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  7. 2024Direct Preference Optimization with an Offset
  8. 2024sDPO: Don't Use Your Data All at Once