mlsanityData to 14 Oct 2026

Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback

arXiv 2503.08942 ↗First posted 11 Mar 2025cs.LGcited by 16

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has become essential for improving language model capabilities, but traditional approaches rely on the assumption that human preferences follow a transitive Bradley-Terry model. This assumption fails to capture the non-transitive nature of populational human preferences. Nash learning from human feedback (NLHF), targeting non-transitive preferences, is a problem of computing the Nash equilibrium (NE) of the two-player constant-sum game defined by the human preference. We introduce Extragradient preference optimization (EGPO), a novel algorithm for NLHF achieving last-iterate linear convergence to the NE of KL-regularized games and polynomial convergence to the NE of original games, while being robust to noise. Unlike previous approaches that rely on nested optimization, we derive an equivalent implementation using gradients of an online variant of the identity preference optimization (IPO) loss, enabling more faithful implementation for neural networks. Our empirical evaluations demonstrate EGPO's superior performance over baseline methods when training for the same number of epochs, as measured by pairwise win-rates using the ground truth preference. These results validate both the theoretical strengths and practical advantages of EGPO for language model alignment with non-transitive human preferences.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 67 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

45 of 67 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent24 Feb
  2. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling6 Feb
  3. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration2 Feb
  4. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment22 Oct
  5. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization11 Oct
  6. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  7. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  8. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  9. 2024Multi-turn Reinforcement Learning from Preference Human Feedback23 May
  10. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  11. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  12. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
8 more
  1. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  3. 2023Nash Learning from Human Feedback1 Dec
  4. 2023Is RLHF More Difficult than Standard RL?25 Jun
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  10. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2013Optimization, Learning, and Games with Predictable Sequencescited by 188
13 more
  1. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  2. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  3. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  4. 2018Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimizationcited by 98
  5. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  6. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  7. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  8. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
  9. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  10. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  11. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
  12. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  13. 2024Advances in Preference-based Reinforcement Learning: A Reviewcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  2. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 10
  3. 2025Multiplayer Nash Preference Optimizationcited by 7
  4. 2025ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstructioncited by 5
  5. 2025Proximal Point Nash Learning from Human Feedbackcited by 5
  6. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  7. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedbackcited by 0
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  9. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scalingcited by 0
  10. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibriumcited by 0
  11. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignmentcited by 0
  12. 2026Bilevel Optimization over Saddle Points of Zero-Sum Markov Gamescited by 0
4 more
  1. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  2. 2026Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learningcited by 0
  3. 2026Inference-Time Nash Alignmentcited by 0
  4. 2026Robust Nash Alignment under Preference Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees
  4. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  5. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  7. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF