mlsanityData to 7 Oct 2026

Inference-Time Nash Alignment

arXiv 2609.08082 ↗First posted 8 Sep 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the model parameters which are not provided by many state-of-the art models. Inference-time alignment offers a cost-effective alternative without updating model parameters. However, existing inference-time methods rely on a scalar reward model derived under a Bradley-Terry assumption, which cannot represent general preferences. Following recent work on fine-tuning with generalized preferences, in this work, we initiate the study of inference-time alignment under general preferences. We formulate the problem as obtaining a Nash equilibrium of a two-player zero-sum game between policies. We propose two algorithms: Best-of-Nash (BoN) and Nash Mirror Descent (NMD). We prove that both algorithms achieve a duality gap that matches the problem lower bound. Empirically, we implement the two methods on three datasets, which shows that our methods substantially outperform the base policy, converging to the performance of the fine-tuned models. Moreover, our results show that NMD remains robust across the regularization parameter.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 49 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

36 of 49 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback11 Mar
  2. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent24 Feb
  3. 2025Jackpot! Alignment as a Maximal Lottery31 Jan
  4. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  5. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  7. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  9. 2023Nash Learning from Human Feedback1 Dec
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  9. 2013Optimization, Learning, and Games with Predictable Sequencescited by 187
  10. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  11. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  12. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
11 more
  1. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  2. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  3. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  4. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  5. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  6. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  7. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  8. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  9. 2023The Sample Complexity of Approximate Rejection Sampling with Applications to Smoothed Online Learningcited by 13
  10. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  11. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Robust Nash Alignment under Preference Uncertainty
  2. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback
  3. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  4. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  5. 2025Multiplayer Nash Preference Optimization
  6. 2026Efficient Exploration for Iterative Nash Preference Optimization
  7. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
  8. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment