mlsanityData to 14 Oct 2026

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

arXiv 2309.16240 ↗First posted 28 Sep 2023cs.LG cs.AI stat.MLcited by 174

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The increasing capabilities of large language models (LLMs) raise opportunities for artificial general intelligence but concurrently amplify safety concerns, such as potential misuse of AI systems, necessitating effective AI alignment. Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising pathway towards AI alignment but brings forth challenges due to its complexity and dependence on a separate reward model. Direct Preference Optimization (DPO) has been proposed as an alternative, and it remains equivalent to RLHF under the reverse KL regularization constraint. This paper presents f-DPO, a generalized approach to DPO by incorporating diverse divergence constraints. We show that under certain f-divergences, including Jensen-Shannon divergence, forward KL divergences and α-divergences, the complex relationship between the reward and optimal policy can also be simplified by addressing the Karush-Kuhn-Tucker conditions. This eliminates the need for estimating the normalizing constant in the Bradley-Terry model and enables a tractable mapping between the reward function and the optimal policy. Our approach optimizes LLMs to align with human preferences in a more efficient and supervised manner under a broad set of divergence constraints. Empirically, adopting these divergences ensures a balance between alignment performance and generation diversity. Importantly, f-DPO outperforms PPO-based methods in divergence efficiency, and divergence constraints directly influence expected calibration error (ECE).

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 48 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

37 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  2. 2022Training language models to follow instructions with human feedback4 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
22 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2023Whose Opinions Do Language Models Reflect?cited by 773
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  12. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  13. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  14. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  15. 2018AI safety via debatecited by 400
  16. 2023Pretraining Language Models with Human Preferencescited by 247
  17. 2023An Overview of Catastrophic AI Riskscited by 222
  18. 2023Model evaluation for extreme riskscited by 177
  19. 2018Supervising strong learners by amplifying weak expertscited by 145
  20. 2020A Distributional Approach to Controlled Text Generationcited by 119
  21. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  22. 2023Reward Collapse in Aligning Large Language Modelscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 61; 2025: 75; 2026: 37.

174 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  4. 2024Token-level Direct Preference Optimizationcited by 135
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  6. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  7. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  8. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 93
  9. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  10. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  11. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  12. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
38 more
  1. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  2. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  3. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
  4. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  5. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  6. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  7. 2024Learn Your Reference Model for Real Good Alignmentcited by 57
  8. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 53
  9. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  10. 2025Diverse Preference Optimizationcited by 52
  11. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  12. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  13. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  14. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  15. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  16. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  17. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  18. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  19. 2025A Survey of Direct Preference Optimizationcited by 36
  20. 2024Prompt Optimization with Human Feedbackcited by 35
  21. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  22. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  23. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  24. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  25. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  26. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  27. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  28. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  29. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  30. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  31. 2024Filtered Direct Preference Optimizationcited by 24
  32. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  33. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  34. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 23
  35. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  36. 2024Self-Improving Robust Preference Optimizationcited by 19
  37. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimizationcited by 19
  38. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersionscited by 18

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Principled Loss Function for Direct Language Model Alignment
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  3. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization
  4. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  5. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  6. 2024Towards Efficient Exact Optimization of Language Model Alignment
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024Accelerated Preference Optimization for Large Language Model Alignment