mlsanityData to 7 Oct 2026

Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning

arXiv 2407.00617 ↗First posted 30 Jun 2024cs.LG cs.AI cs.CLcited by 45

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) has achieved great success in aligning large language models (LLMs) with human preferences. Prevalent RLHF approaches are reward-based, following the Bradley-Terry (BT) model assumption, which may not fully capture the complexity of human preferences. In this paper, we explore RLHF under a general preference framework and approach it from a game-theoretic perspective. Specifically, we formulate the problem as a two-player game and propose a novel online algorithm, iterative Nash policy optimization (INPO). The key idea is to let the policy play against itself via no-regret learning, thereby approximating the Nash policy. Unlike previous methods, INPO bypasses the need for estimating the expected win rate for individual responses, which typically incurs high computational or annotation costs. Instead, we introduce a new loss objective that is directly minimized over a preference dataset. We provide theoretical analysis for our approach and demonstrate its effectiveness through experiments on various representative benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 42.6% length-controlled win rate on AlpacaEval 2.0 and a 37.8% win rate on Arena-Hard, showing substantial improvement over the state-of-the-art online RLHF algorithms.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 58 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

45 of 58 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  7. 2023Nash Learning from Human Feedback1 Dec
  8. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  9. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsal18 Sep
  10. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
4 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  2. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
17 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2013Optimization, Learning, and Games with Predictable Sequencescited by 187
  3. 2015Fast Convergence of Regularized Learning in Gamescited by 149
  4. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  5. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
  6. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  7. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
  8. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  9. 2021Near-Optimal No-Regret Learning in General Gamescited by 85
  10. 2021When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?cited by 63
  11. 2020Hedging in games: Faster convergence of external and swap regretscited by 57
  12. 2021Provably Efficient Reinforcement Learning in Decentralized General-Sum Markov Gamescited by 42
  13. 2022No-Regret Learning in Time-Varying Zero-Sum Gamescited by 36
  14. 2022Near-Optimal Learning of Extensive-Form Games with Imperfect Informationcited by 26
  15. 2021Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recallcited by 21
  16. 2022Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descentcited by 15
  17. 2022Adapting to game trees in zero-sum imperfect information gamescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 20; 2026: 19.

45 papers.

Sort:

By area:

  1. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  2. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  3. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 28
  4. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  5. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  6. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  7. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  8. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  9. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  10. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  11. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  12. 2025Doubly Robust Alignment for Large Language Modelscited by 11
33 more
  1. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
  2. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  3. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 9
  4. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 9
  5. 2024Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimizationcited by 8
  6. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8
  7. 2025Multiplayer Nash Preference Optimizationcited by 7
  8. 2026Beyond Pessimism: Offline Learning in KL-regularized Gamescited by 7
  9. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Modelcited by 6
  10. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  11. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  12. 2026Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspectivecited by 4
  13. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
  14. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guaranteescited by 3
  15. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 3
  16. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 3
  17. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  18. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learningcited by 1
  19. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  20. 2025Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimizationcited by 0
  21. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  22. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedbackcited by 0
  23. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scalingcited by 0
  24. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  25. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibriumcited by 0
  26. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignmentcited by 0
  27. 2026Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problemscited by 0
  28. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  29. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  30. 2026Inference-Time Nash Alignmentcited by 0
  31. 2026Minimax Last-Iterate Convergence in Matrix Games with Observed Actionscited by 0
  32. 2026Multi-LLM Collaborative Alignment via Stackelberg Gamescited by 0
  33. 2026Robust Nash Alignment under Preference Uncertaintycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  2. 2025Proximal Point Nash Learning from Human Feedback
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Multi-turn Reinforcement Learning from Preference Human Feedback
  5. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  6. 2024Language Alignment via Nash-learning and Adaptive feedback
  7. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  8. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium