# WPO: Enhancing RLHF with Weighted Preference Optimization

arXiv: https://arxiv.org/abs/2406.11827 · HTML: https://mlsanity.com/paper/2406.11827/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 36)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study](https://mlsanity.com/paper/2404.10719/) (2024)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Self-Play Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2405.00675/) (2024)
- [Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data](https://mlsanity.com/paper/2404.14367/) (2024)
- [Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences](https://mlsanity.com/paper/2404.03715/) (2024)
- [Generalized Preference Optimization: A Unified Approach to Offline Alignment](https://mlsanity.com/paper/2402.05749/) (2024)
- [Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://mlsanity.com/paper/2402.18571/) (2024)

## Built on by (most cited first, 30 of 54)

- [Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?](https://mlsanity.com/paper/2502.00674/) (2025)
- [The Best Instruction-Tuning Data are Those That Fit](https://mlsanity.com/paper/2502.04194/) (2025)
- [From Lists to Emojis: How Format Bias Affects Model Alignment](https://mlsanity.com/paper/2409.11704/) (2024)
- [Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs](https://mlsanity.com/paper/2506.17080/) (2025)
- [Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples](https://mlsanity.com/paper/2502.09650/) (2025)
- [RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization](https://mlsanity.com/paper/2410.04203/) (2024)
- [T-REG: Preference Optimization with Token-Level Reward Regularization](https://mlsanity.com/paper/2412.02685/) (2024)
- [Robust Preference Optimization via Dynamic Target Margins](https://mlsanity.com/paper/2506.03690/) (2025)
- [Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.22995/) (2026)
- [Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning](https://mlsanity.com/paper/2504.04524/) (2025)
- [Pluralistic Off-policy Evaluation and Alignment](https://mlsanity.com/paper/2509.19333/) (2025)
- [Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment](https://mlsanity.com/paper/2502.00203/) (2025)
- [Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model](https://mlsanity.com/paper/2412.13862/) (2024)
- [Aligning Large Language Models with Implicit Preferences from User-Generated Content](https://mlsanity.com/paper/2506.04463/) (2025)
- [Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs](https://mlsanity.com/paper/2506.10054/) (2025)
- [When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2605.07260/) (2026)
- [Length Desensitization in Direct Preference Optimization](https://mlsanity.com/paper/2409.06411/) (2024)
- [Efficient Construction of Model Family through Progressive Training Using Model Expansion](https://mlsanity.com/paper/2504.00623/) (2025)
- [Multiplayer Nash Preference Optimization](https://mlsanity.com/paper/2509.23102/) (2025)
- [FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion](https://mlsanity.com/paper/2503.04222/) (2025)
- [Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2507.15507/) (2025)
- [SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins](https://mlsanity.com/paper/2410.09362/) (2024)
- [Advantage-Guided Distillation for Preference Alignment in Small Language Models](https://mlsanity.com/paper/2502.17927/) (2025)
- [Entropy-Based Adaptive Weighting for Self-Training](https://mlsanity.com/paper/2503.23913/) (2025)
- [One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generation](https://mlsanity.com/paper/2602.02033/) (2026)
- [InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization](https://mlsanity.com/paper/2503.15880/) (2025)
- [TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation](https://mlsanity.com/paper/2511.15767/) (2025)
- [Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment](https://mlsanity.com/paper/2601.17329/) (2026)
- [FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings](https://mlsanity.com/paper/2501.06645/) (2025)
- [SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization](https://mlsanity.com/paper/2511.06222/) (2025)

## Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a promising solution to align large language models (LLMs) more closely with human values. Off-policy preference optimization, where the preference data is obtained from other models, is widely adopted due to its cost efficiency and scalability. However, off-policy preference optimization often suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization. In this paper, we propose a novel strategy to mitigate this problem by simulating on-policy learning with off-policy preference data. Our Weighted Preference Optimization (WPO) method adapts off-policy data to resemble on-policy data more closely by reweighting preference pairs according to their probability under the current policy. This method not only addresses the distributional gap problem but also enhances the optimization process without incurring additional costs. We validate our method on instruction following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 but also establishes a remarkable length-controlled winning rate against GPT-4-turbo of 76.7% based on Gemma-2-9b-it. We release the code and models at https://github.com/wzhouad/WPO.