mlsanityData to 7 Oct 2026

Aligning Language Models with Preferences through f-divergence Minimization

arXiv 2302.08215 ↗First posted 16 Feb 2023cs.CL cs.LG stat.MLcited by 118

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models with preferences can be posed as approximating a target distribution representing some desired behavior. Existing approaches differ both in the functional form of the target distribution and the algorithm used to approximate it. For instance, Reinforcement Learning from Human Feedback (RLHF) corresponds to minimizing a reverse KL from an implicit target distribution arising from a KL penalty in the objective. On the other hand, Generative Distributional Control (GDC) has an explicit target distribution and minimizes a forward KL from it using the Distributional Policy Gradient (DPG) algorithm. In this paper, we propose a new approach, f-DPG, which allows the use of any f-divergence to approximate any target distribution that can be evaluated. f-DPG unifies both frameworks (RLHF, GDC) and the approximation methods (DPG, RL with KL penalties). We show the practical benefits of various choices of divergence objectives and demonstrate that there is no universally optimal objective but that different divergences present different alignment and diversity trade-offs. We show that Jensen-Shannon divergence strikes a good balance between these objectives, and frequently outperforms forward KL divergence by a wide margin, leading to significant improvements over prior work. These distinguishing characteristics between divergences persist as the model size increases, highlighting the importance of selecting appropriate divergence objectives.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 129 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

66 of 129 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 64

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2018Hierarchical Neural Story Generationcited by 1,182
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  11. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  12. 2018Which Training Methods for GANs do actually Converge?cited by 672
  13. 2020Recipes for building an open-domain chatbotcited by 593
  14. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  15. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  16. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  17. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  18. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  19. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  20. 2022Self-critiquing models for assisting human evaluatorscited by 372
  21. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  22. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  23. 2015A note on the evaluation of generative modelscited by 291
  24. 2022Teaching language models to support answers with verified quotescited by 290
  25. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  26. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  27. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  28. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  29. 2021Challenges in Detoxifying Language Modelscited by 169
  30. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  31. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  32. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  33. 2016Mode Regularized Generative Adversarial Networkscited by 123
  34. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  35. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  36. 2020A Distributional Approach to Controlled Text Generationcited by 117
  37. 2022Language Model Cascadescited by 104
  38. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 36; 2025: 40; 2026: 18.

118 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  5. 2023Large Language Model Alignment: A Surveycited by 241
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  7. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  9. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  10. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  11. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  12. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
38 more
  1. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  2. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  3. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  4. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  5. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  6. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  7. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  8. 2023Learning to Generate Better Than Your LLMcited by 49
  9. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  10. 2023Learning from models beyond fine-tuningcited by 47
  11. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 39
  12. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  13. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 37
  14. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  15. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  16. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 32
  17. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  18. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  19. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  20. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  21. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 27
  22. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27
  23. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectorscited by 26
  24. 2023Compositional preference models for aligning LMscited by 25
  25. 2023Align on the Fly: Adapting Chatbot Behavior to Established Normscited by 22
  26. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  27. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learningcited by 21
  28. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20
  29. 2024Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modelingcited by 20
  30. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  31. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  32. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  33. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
  34. 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Expertscited by 15
  35. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  36. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  37. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  38. 2024Reward Learning From Preference With Tiescited by 13

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  2. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  3. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimization
  4. 2025A Principled Loss Function for Direct Language Model Alignment
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  6. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  7. 2024Accelerated Preference Optimization for Large Language Model Alignment
  8. 2024Towards Efficient Exact Optimization of Language Model Alignment