mlsanityData to 7 Oct 2026

Robust Nash Alignment under Preference Uncertainty

arXiv 2610.00715 ↗First posted 30 Sep 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF also fits: LLM preference optimization methods

Abstract (arXiv)

Preference-based alignment methods typically optimize against a single preference model, and can therefore be brittle when pairwise preferences are uncertain: noisy, heterogeneous, or shift after deployment. To address these issues, we propose Robust Nash Alignment, a game-theoretic framework for alignment to uncertain pairwise preferences. Our formulation has a major learner seeking a policy with a large worst-case win rate against both an adversarial competitor and any preference kernel lying in an ambiguity set around a nominal preference. When the ambiguity set captures the uncertainty in preferences, the resulting robust objective of the game directly yields a certified lower bound on worst-case performance. However, we note this problem is computationally challenging to optimize, and to address this, we introduce a four-player primal-dual proxy game involving the leader policy, follower policy, adversarial kernel, and dual variable, and develop a single-loop optimistic mirror descent-ascent algorithm for it. We show that the proxy always lower-bounds the truncated hard-constrained objective, quantify the proxy-to-hard gap, and characterize an exactness condition under which the proxy recovers the robust objective. We then prove an (O(1/√T)) average-iteration convergence for the proxy-game duality gap, which implies a near-optimal robust policy for the original robust objective. Experiments on controlled tabular games and LLM alignment with uncertain preference further validate the convergence theory and show improved performance over nominal baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 54 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

39 of 54 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game18 Dec
  2. 2025Proximal Point Nash Learning from Human Feedback26 May
  3. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback11 Mar
  4. 2025Distributionally Robust Reinforcement Learning with Human Feedback1 Mar
  5. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  6. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedback4 Jun
  7. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  8. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  9. 2023Nash Learning from Human Feedback1 Dec
  10. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
2 more
  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  11. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  12. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
13 more
  1. 2015Fast Convergence of Regularized Learning in Gamescited by 149
  2. 2019A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approachcited by 142
  3. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 132
  4. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  5. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  7. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  8. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  9. 2025Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Gamescited by 11
  10. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
  11. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 9
  12. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  13. 2022Computing the optimal distributionally-robust strategy to commit tocited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Inference-Time Nash Alignment
  2. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  3. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  4. 2026Efficient Exploration for Iterative Nash Preference Optimization
  5. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  6. 2025Multiplayer Nash Preference Optimization
  7. 2026Groupwise Distortion Guarantees for Preference-Based Alignment
  8. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees