mlsanityData to 7 Oct 2026

Generalized Preference Optimization: A Unified Approach to Offline Alignment

arXiv 2402.05749 ↗First posted 8 Feb 2024cs.LG cs.AI stat.MLcited by 155

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 31 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

17 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
3 more
  1. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  2. 2020VarGrad: A Low-Variance Gradient Estimator for Variational Inferencecited by 73
  3. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 62; 2025: 72; 2026: 21.

155 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  4. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  5. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  6. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  7. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  8. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  9. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  10. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  11. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  12. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
38 more
  1. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  2. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  3. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  4. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  5. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  6. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  7. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  8. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  9. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  10. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  11. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  12. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  13. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  14. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  15. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  16. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  17. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  18. 2024Prompt Optimization with Human Feedbackcited by 35
  19. 2025A Survey of Direct Preference Optimizationcited by 35
  20. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  21. 2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Financecited by 32
  22. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  23. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  24. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  25. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  26. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  27. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  28. 2024Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimizationcited by 25
  29. 2024Filtered Direct Preference Optimizationcited by 24
  30. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  31. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  32. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23
  33. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Studycited by 22
  34. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  35. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  36. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  37. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 18
  38. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersionscited by 18

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  2. 2025Linear Preference Optimization: Decoupled Gradient Control via Absolute Regularization
  3. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimization
  4. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
  5. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
  6. 2024Distributional Preference Alignment of LLMs via Optimal Transport
  7. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences
  8. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration