mlsanityData to 14 Oct 2026

Preference as Reward, Maximum Preference Optimization with Importance Sampling

arXiv 2312.16430 ↗First posted 27 Dec 2023cs.LG cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference learning is a key technology for aligning language models with human values. Reinforcement Learning from Human Feedback (RLHF) is a model-based algorithm to optimize preference learning, which first fits a reward model for preference scores and then optimizes the generating policy with an on-policy PPO algorithm to maximize the reward. The processing of RLHF is complex, time-consuming, and unstable. The Direct Preference Optimization (DPO) algorithm uses an off-policy algorithm to directly optimize the generating policy and eliminates the need for a reward model. DPO is more data-efficient and stable. However, DPO has a drawback of overfitting to the preference data and ignoring the KL-regularization term when the preference is deterministic. Identity mapping Preference Optimization(IPO) uses a root-finding MSE loss to incorporate KL-regularization. However, both DPO and IPO fail to properly address the KL-regularization term because the support of the preference distribution is not equal to the reference distribution. In this paper, we propose a simple and intuitive off-policy preference optimization algorithm from an importance sampling view, which we call Maximum Preference Optimization (MPO). MPO incorporates the off-policy KL-regularization term, making regularization truly effective. MPO achieves the best of both worlds by combining the objectives of RLHF and IPO while being an off-policy algorithm. Furthermore, MPO eliminates the need for a reward model and reference policy, simplifying the learning process and reducing memory usage.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 28 references that are in the map. First posted on arXiv 27 Dec 2023.

Builds on

24 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
10 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  8. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  9. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  10. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  3. 2025Verifiable Format Control for Large Language Model Generationscited by 17
  4. 2026Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Filtered Direct Preference Optimization
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2024New Desiderata for Direct Preference Optimization
  4. 2024WPO: Enhancing RLHF with Weighted Preference Optimization
  5. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  6. 2024Minor DPO reject penalty to increase training robustness
  7. 2024Active Preference Learning for Large Language Models
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences