mlsanityData to 13 Oct 2026

f-PO: Generalizing Preference Optimization with f-divergence Minimization

arXiv 2410.21662 ↗First posted 29 Oct 2024cs.CL cs.LGcited by 19

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization has made significant progress recently, with numerous methods developed to align language models with human preferences. This paper introduces f-divergence Preference Optimization (f-PO), a novel framework that generalizes and extends existing approaches. f-PO minimizes f-divergences between the optimized policy and the optimal policy, encompassing a broad family of alignment methods using various divergences. Our approach unifies previous algorithms like DPO and EXO, while offering new variants through different choices of f-divergences. We provide theoretical analysis of f-PO's properties and conduct extensive experiments on state-of-the-art language models using benchmark datasets. Results demonstrate f-PO's effectiveness across various tasks, achieving superior performance compared to existing methods on popular benchmarks such as AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2. Additionally, we present ablation studies exploring the impact of different f-divergences, offering insights into the trade-offs between regularization and performance in offline preference optimization. Our work contributes both practical algorithms and theoretical understanding to the field of language model alignment. Code is available at https://github.com/MinkaiXu/fPO.

Key facts. Cited by 19 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 40 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

27 of 40 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  9. 2023Preference Ranking Optimization for Human Alignment30 Jun
  10. 2022Training language models to follow instructions with human feedback4 Mar
  11. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  5. 2017Deep reinforcement learning from human preferencescited by 3,871
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  7. 2020Learning to summarize from human feedbackcited by 2,575
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,068
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 877
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  12. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 505
4 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 491
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 459
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 443
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 400

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 13; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  2. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 12
  3. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimizationcited by 9
  4. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  5. 2025Divergence Minimization Preference Optimization for Diffusion Model Alignmentcited by 8
  6. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  7. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  8. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  9. 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
  10. 2025Preference learning made easy: Everything should be understood through win ratecited by 3
  11. 2026f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Datacited by 3
  12. 2025C2-DPO: Constrained Controlled Direct Preference Optimizationcited by 2
7 more
  1. 2026Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignmentcited by 2
  2. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analysescited by 2
  3. 2025EPO: Diverse and Realistic Protein Ensemble Generation via Energy Preference Optimizationcited by 1
  4. 2026FlowCPO: A Unified Divergence View of Preference Alignment for Flow Modelscited by 1
  5. 2025Symmetric Behavior Regularized Policy Optimizationcited by 0
  6. 2025Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficultycited by 0
  7. 2026Displacement-Resistant Extensions of DPO with Nonconvex f-Divergencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligning Language Models with Preferences through f-divergence Minimization
  2. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  3. 2025DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization
  4. 2025Robust Preference Optimization via Dynamic Target Margins
  5. 2026GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
  6. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling