# f-PO: Generalizing Preference Optimization with f-divergence Minimization

arXiv: https://arxiv.org/abs/2410.21662 · HTML: https://mlsanity.com/paper/2410.21662/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [Statistical Rejection Sampling Improves Preference Optimization](https://mlsanity.com/paper/2309.06657/) (2023)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024)
- [Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints](https://mlsanity.com/paper/2309.16240/) (2023)
- [LiPO: Listwise Preference Optimization through Learning-to-Rank](https://mlsanity.com/paper/2402.01878/) (2024)

## Built on by (most cited first, 19 of 19)

- [The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward](https://mlsanity.com/paper/2509.07430/) (2025)
- [Discrete Diffusion Trajectory Alignment via Stepwise Decomposition](https://mlsanity.com/paper/2507.04832/) (2025)
- [Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization](https://mlsanity.com/paper/2502.16825/) (2025)
- [Preference Optimization by Estimating the Ratio of the Data Distribution](https://mlsanity.com/paper/2505.19601/) (2025)
- [Divergence Minimization Preference Optimization for Diffusion Model Alignment](https://mlsanity.com/paper/2507.07510/) (2025)
- [Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator](https://mlsanity.com/paper/2503.15457/) (2025)
- [Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences](https://mlsanity.com/paper/2602.23116/) (2026)
- [Less is More: Improving LLM Alignment via Preference Data Selection](https://mlsanity.com/paper/2502.14560/) (2025)
- [f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Data](https://mlsanity.com/paper/2605.15417/) (2026)
- [The Differences Between Direct Alignment Algorithms are a Blur](https://mlsanity.com/paper/2502.01237/) (2025)
- [Preference learning made easy: Everything should be understood through win rate](https://mlsanity.com/paper/2502.10505/) (2025)
- [C2-DPO: Constrained Controlled Direct Preference Optimization](https://mlsanity.com/paper/2502.17507/) (2025)
- [Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment](https://mlsanity.com/paper/2602.01685/) (2026)
- [f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses](https://mlsanity.com/paper/2605.06977/) (2026)
- [EPO: Diverse and Realistic Protein Ensemble Generation via Energy Preference Optimization](https://mlsanity.com/paper/2511.10165/) (2025)
- [FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models](https://mlsanity.com/paper/2609.09905/) (2026)
- [Symmetric Behavior Regularized Policy Optimization](https://mlsanity.com/paper/2508.04225/) (2025)
- [Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty](https://mlsanity.com/paper/2510.05534/) (2025)
- [Displacement-Resistant Extensions of DPO with Nonconvex f-Divergences](https://mlsanity.com/paper/2602.06788/) (2026)

## Abstract (arXiv)

Preference optimization has made significant progress recently, with numerous methods developed to align language models with human preferences. This paper introduces f-divergence Preference Optimization (f-PO), a novel framework that generalizes and extends existing approaches. f-PO minimizes f-divergences between the optimized policy and the optimal policy, encompassing a broad family of alignment methods using various divergences. Our approach unifies previous algorithms like DPO and EXO, while offering new variants through different choices of f-divergences. We provide theoretical analysis of f-PO's properties and conduct extensive experiments on state-of-the-art language models using benchmark datasets. Results demonstrate f-PO's effectiveness across various tasks, achieving superior performance compared to existing methods on popular benchmarks such as AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2. Additionally, we present ablation studies exploring the impact of different f-divergences, offering insights into the trade-offs between regularization and performance in offline preference optimization. Our work contributes both practical algorithms and theoretical understanding to the field of language model alignment. Code is available at https://github.com/MinkaiXu/fPO.