mlsanityData to 14 Oct 2026

Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment

arXiv 2410.16714 ↗First posted 22 Oct 2024cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 68 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

46 of 68 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024BOND: Aligning LLMs with Best-of-N Distillation19 Jul
  2. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  4. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  7. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  8. 2023Nash Learning from Human Feedback1 Dec
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  10. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
2 more
  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
20 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  6. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  7. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
  8. 2024Token-level Direct Preference Optimizationcited by 135
  9. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  10. 2017Cycles in adversarial regularized learningcited by 124
  11. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  12. 2020Mirror Descent Policy Optimizationcited by 106
  13. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  14. 2016Learning in games with continuous action sets and unknown payoff functionscited by 81
  15. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  16. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  17. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  18. 2015Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issuescited by 32
  19. 2023Evaluating Agents using Social Choice Theorycited by 11
  20. 2023Min-Max Optimization Made Simple: Approximating the Proximal Point Method via Contraction Mapscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 8.

20 papers.

Sort:

By area:

  1. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  2. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  3. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 16
  4. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 10
  5. 2025Incentivizing Truthful Language Models via Peer Elicitation Gamescited by 9
  6. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8
  7. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 7
  8. 2025Multiplayer Nash Preference Optimizationcited by 7
  9. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  10. 2026Self-Evolving Agents with Anytime-Valid Certificatescited by 6
  11. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  12. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guaranteescited by 3
8 more
  1. 2026Multi-Head Attention Is a Multi-Player Gamecited by 2
  2. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedbackcited by 0
  3. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scalingcited by 0
  4. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  5. 2026Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problemscited by 0
  6. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  7. 2026S-SPPO: Semantic-Calibrated Self-Play Preference Optimizationcited by 0
  8. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  2. 2024Investigating Regularization of Self-Play Language Models
  3. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  4. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  5. 2024Language Alignment via Nash-learning and Adaptive feedback
  6. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment
  7. 2024Accelerated Preference Optimization for Large Language Model Alignment
  8. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration