mlsanityData to 7 Oct 2026

WPO: Enhancing RLHF with Weighted Preference Optimization

arXiv 2406.11827 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 53

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a promising solution to align large language models (LLMs) more closely with human values. Off-policy preference optimization, where the preference data is obtained from other models, is widely adopted due to its cost efficiency and scalability. However, off-policy preference optimization often suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization. In this paper, we propose a novel strategy to mitigate this problem by simulating on-policy learning with off-policy preference data. Our Weighted Preference Optimization (WPO) method adapts off-policy data to resemble on-policy data more closely by reweighting preference pairs according to their probability under the current policy. This method not only addresses the distributional gap problem but also enhances the optimization process without incurring additional costs. We validate our method on instruction following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 but also establishes a remarkable length-controlled winning rate against GPT-4-turbo of 76.7% based on Gemma-2-9b-it. We release the code and models at https://github.com/wzhouad/WPO.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 47 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

36 of 47 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  3. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels22 Apr
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  7. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  8. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  9. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2022Training language models to follow instructions with human feedback4 Mar
  12. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
12 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2022Prompting GPT-3 To Be Reliablecited by 258
  7. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  9. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  11. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  12. 2022The CRINGE Loss: Learning what language not to modelcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 28; 2026: 17.

53 papers.

Sort:

By area:

  1. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  2. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  3. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  4. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  5. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  6. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  7. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  8. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  9. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoningcited by 12
  10. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  11. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
  12. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
38 more
  1. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  2. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  3. 2024Length Desensitization in Direct Preference Optimizationcited by 8
  4. 2025Efficient Construction of Model Family through Progressive Training Using Model Expansioncited by 8
  5. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
  6. 2025Multiplayer Nash Preference Optimizationcited by 7
  7. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 7
  8. 2025FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusioncited by 6
  9. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedbackcited by 6
  10. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  11. 2025Entropy-Based Adaptive Weighting for Self-Trainingcited by 5
  12. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  13. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
  14. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 4
  15. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankingscited by 3
  16. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 3
  17. 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimizationcited by 3
  18. 2025TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generationcited by 3
  19. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3
  20. 2024VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedbackcited by 2
  21. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  22. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  23. 2025Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimizationcited by 1
  24. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Onlycited by 1
  25. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  26. 2026Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitationcited by 1
  27. 2026Margin-Adaptive Confidence Ranking for Reliable LLM Judgementcited by 1
  28. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  29. 2026Normalized Rewards for Preference Optimizationcited by 1
  30. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awarenesscited by 0
  31. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0
  32. 2025Preference Distillation via Value based Reinforcement Learningcited by 0
  33. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  34. 2025MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generationcited by 0
  35. 2025RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedbackcited by 0
  36. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  37. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Strongercited by 0
  38. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferencescited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerated Preference Optimization for Large Language Model Alignment
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  3. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  5. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  6. 2024Filtered Direct Preference Optimization
  7. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  8. 2023Preference as Reward, Maximum Preference Optimization with Importance Sampling