mlsanityData to 14 Oct 2026

Efficient Exploration for Iterative Nash Preference Optimization

arXiv 2606.01382 ↗First posted 31 May 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference alignment is central to improving large language models (LLMs), but reward-based formulations can be restrictive when human preferences are non-transitive. Nash learning from human feedback (NLHF) addresses this limitation by modeling alignment as a preference game and seeking a Nash equilibrium. However, the learning-theoretic foundations of scalable NLHF remain limited: existing regret guarantees rely on explicit preference-model estimation and minimax oracles, whereas simpler iterative methods lack such guarantees. We study online iterative NLHF and identify exploration as a key obstacle. First, we show that standard iterative NLHF can incur an exponential dependence on the inverse KL-regularization parameter, demonstrating that implicit exploration through policy updates can be insufficient. We then propose Exploratory Nash Preference Optimization (ENPO), which combines a SFT-type regularization with adversarial policy exploration. ENPO eliminates this exponential dependence without requiring minimax oracles or explicit preference-model estimation. We further introduce Bonus-Explorer ENPO (BENPO), which uses additional oracles to achieve an O(log T) regret bound. Finally, we develop Direct ENPO (DENPO), a practical variant of ENPO for fine-tuning LLMs. Experiments with Llama-3-8B-Instruct demonstrate consistent improvements over the evaluated RLHF and NLHF baselines across multiple benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 105 of its 121 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

105 of 121 references.

Closest ancestors · 41

Same direction, newest first

  1. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences26 Feb
  2. 2025Greedy Sampling Is Provably Efficient for RLHF28 Oct
  3. 2025Multiplayer Nash Preference Optimization27 Sep
  4. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching27 May
  5. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium14 Mar
  6. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback11 Mar
  7. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent24 Feb
  8. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learning11 Feb
  9. 2025Jackpot! Alignment as a Maximal Lottery31 Jan
  10. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment22 Oct
  11. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  12. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
29 more
  1. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  4. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning30 Apr
  5. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  6. 2024Mapping Social Choice Theory to RLHF19 Apr
  7. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  8. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  9. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  10. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  11. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  12. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  13. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  14. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  15. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  16. 2023Nash Learning from Human Feedback1 Dec
  17. 2023Demonstration-Regularized RL26 Oct
  18. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  19. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  20. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  21. 2023Is RLHF More Difficult than Standard RL?25 Jun
  22. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  23. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  24. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  25. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  26. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  27. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  28. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  29. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  12. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
38 more
  1. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  2. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  5. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  7. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  8. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  9. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  10. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  11. 2017A unified view of entropy-regularized Markov decision processescited by 182
  12. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  13. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  14. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  15. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  16. 2016Variance-based regularization with convex objectivescited by 155
  17. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  18. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  19. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  20. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  21. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  22. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  23. 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 92
  24. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  25. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  26. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  27. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  28. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  29. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  30. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  31. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  32. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  33. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  34. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  35. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  36. 2024Efficient Exploration for LLMscited by 45
  37. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  38. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34

50 most cited of 64 All 64 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback
  2. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  3. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
  4. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  5. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  6. 2025Proximal Point Nash Learning from Human Feedback
  7. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  8. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibrium