mlsanityData to 7 Oct 2026

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

arXiv 2502.16852 ↗First posted 24 Feb 2025cs.LG cs.AI cs.CLcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an O(T-1) bound on the duality gap, improving upon the previous O(T-1/2) result. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 68 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

49 of 68 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees18 Feb
  2. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences30 Oct
  3. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  4. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  5. 2024Multi-turn Reinforcement Learning from Preference Human Feedback23 May
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  7. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  9. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  10. 2024Self-Rewarding Language Models18 Jan
  11. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  12. 2023Nash Learning from Human Feedback1 Dec
6 more
  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  2. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsal18 Sep
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
19 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  4. 2013Optimization, Learning, and Games with Predictable Sequencescited by 187
  5. 2015Fast Convergence of Regularized Learning in Gamescited by 149
  6. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  7. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
  8. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
  9. 2020A Sharp Analysis of Model-based Reinforcement Learning with Self-Playcited by 104
  10. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  11. 2021Near-Optimal No-Regret Learning in General Gamescited by 85
  12. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  13. 2017Online Reinforcement Learning in Stochastic Gamescited by 67
  14. 2020Hedging in games: Faster convergence of external and swap regretscited by 57
  15. 2021Provably Efficient Reinforcement Learning in Decentralized General-Sum Markov Gamescited by 42
  16. 2021Last-iterate Convergence in Extensive-Form Gamescited by 27
  17. 2022Near-Optimal Learning of Extensive-Form Games with Imperfect Informationcited by 26
  18. 2021Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recallcited by 21
  19. 2019Online and Bandit Algorithms for Nonstationary Stochastic Saddle-Point Optimizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 15.

24 papers.

Sort:

By area:

  1. 2025Reinforcement Learning from Human Feedbackcited by 53
  2. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 28
  3. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  4. 2025Harnessing Consistency for Robust Test-Time LLM Ensemblecited by 10
  5. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
  6. 2025Multiplayer Nash Preference Optimizationcited by 7
  7. 2026Beyond Pessimism: Offline Learning in KL-regularized Gamescited by 7
  8. 2025Safety Alignment of LMs via Non-cooperative Gamescited by 6
  9. 2026Subspace Alignment for Vision-Language Model Test-time Adaptationcited by 5
  10. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  11. 2025Proximal Point Nash Learning from Human Feedbackcited by 4
  12. 2026AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMscited by 4
12 more
  1. 2026Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspectivecited by 4
  2. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 3
  3. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 3
  4. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learningcited by 1
  5. 2026One Model, Many Minds: Unlocking Multi-Agent Synergy in a Single Agent via Mixture of Rolescited by 1
  6. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedbackcited by 0
  7. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibriumcited by 0
  8. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignmentcited by 0
  9. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  10. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  11. 2026Inference-Time Nash Alignmentcited by 0
  12. 2026Minimax Last-Iterate Convergence in Matrix Games with Observed Actionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  2. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  3. 2025Online Preference Alignment for Language Models via Count-based Exploration
  4. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  5. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment
  6. 2026Spectral Souping: A Unified Framework for Online Preference Alignment
  7. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  8. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium