mlsanityData to 7 Oct 2026

Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

arXiv 2404.14723 ↗First posted 23 Apr 2024cs.CLcited by 35

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

This study evaluates Direct Preference Optimization (DPO) and its variants for aligning Large Language Models (LLMs) with human preferences, testing three configurations: (1) with Supervised Fine Tuning (SFT), (2) without SFT, and (3) without SFT but using an instruction tuned model. We further investigate how training set size influences model performance. Our evaluation spans 13 benchmarks covering dialogue, reasoning, mathematical problem-solving, question answering, truthfulness, MT-Bench, Big Bench, and the Open LLM Leaderboard. We find that: (1) alignment methods often achieve near optimal performance even with smaller subsets of training data; (2) although they offer limited improvements on complex reasoning tasks, they enhance mathematical problem-solving; and (3) using an instruction tuned model improves truthfulness. These insights highlight the conditions under which alignment methods excel, as well as their limitations.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 46 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

32 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
15 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  13. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  14. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  15. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 14; 2026: 6.

35 papers.

Sort:

By area:

  1. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  4. 2025A Survey of Direct Preference Optimizationcited by 35
  5. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  7. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  8. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  9. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  10. 2024PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinkingcited by 17
  11. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
  12. 2024PORT: Preference Optimization on Reasoning Tracescited by 6
23 more
  1. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Losscited by 6
  2. 2025Gradient Imbalance in Direct Preference Optimizationcited by 6
  3. 2024Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Preference Optimizationcited by 5
  4. 2024TODO: Enhancing LLM Alignment with Ternary Preferencescited by 5
  5. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5
  6. 2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
  7. 2025Entropy-Based Adaptive Weighting for Self-Trainingcited by 5
  8. 2025Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuningcited by 4
  9. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  10. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4
  11. 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
  12. 2025Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350Mcited by 3
  13. 2025What Is Preference Optimization Doing, and Why?cited by 2
  14. 2026Knowledge Graph-Assisted LLM Post-Training for Enhanced Legal Reasoningcited by 2
  15. 2026Mesh-Pro: Asynchronous Advantage-guided Ranking Preference Optimization for Artist-style Quadrilateral Mesh Generationcited by 2
  16. 2026Federated Personal Knowledge Graph Completion with Lightweight Large Language Models for Personalized Recommendationscited by 2
  17. 2026Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversionscited by 2
  18. 2025Improved Algorithms for Differentially Private Language Model Alignmentcited by 1
  19. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignmentcited by 1
  20. 2026Benchmarking Direct Preference Optimization for Medical Large Vision-Language Modelscited by 1
  21. 2025Dual Caption Preference Optimization for Diffusion Modelscited by 0
  22. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimizationcited by 0
  23. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  3. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  4. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward
  8. 2024sDPO: Don't Use Your Data All at Once