mlsanityData to 7 Oct 2026

A Markov Chain Approach to Preference Alignment

arXiv 2606.22652 ↗First posted 21 Jun 2026cs.LG stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We propose Markov Chain from Human Feedback (MCHF), an elementary approach for aligning generative models from pairwise human preferences. Unlike Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar reward, and Nash Learning from Human Feedback (NLHF), which preserves pairwise utilities through a KL-regularized minimax optimization, MCHF uses pairwise preferences directly to define a transition mechanism over model outputs. Given a pairwise utility U(x,y), which quantifies human preference for y over x, and a reference probability distribution μ_mathsfref, we define a Markov kernel mathsfP(x, dy)propto exp(U(x,y))μ_mathsfref(dy), and take the Markov chain starting from μ_mathsfref as an iterative alignment procedure. We show that MCHF converges geometrically fast to the stationary distribution, with a convergence rate governed by the seminorm |U|_oplus=inf_g,f∈ L^∞(μ_mathsfref)|U-goplus f|_∞, which quantifies the non-transitive structure of the pairwise utility. We further show that a mirror-descent algorithm for NLHF satisfies an analogous structure-adaptive convergence guarantee. Finally, through a perturbation analysis, we prove that when |U|_oplus is small, MCHF and NLHF agree up to first order around an RLHF solution, which yields a unified view of reward-based, game-theoretic, and Markovian approaches to alignment. In particular, for two natural algorithms that converge to the MCHF/NLHF equilibria, we show that the first step of MCHF and NLHF recovers the RLHF solution based on the column-sum reward hatf(y)=int μ_mathsfref(dx) U(x, y), and starting from the second iteration, both algorithms incorporate the same linear functional of the residual U-(-hat f)oplus hat f, which captures the non-transitive structure of the pairwise utility U.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 35 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

23 of 35 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game18 Dec
  2. 2025Multiplayer Nash Preference Optimization27 Sep
  3. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching27 May
  4. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium14 Mar
  5. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  6. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  7. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  8. 2023Nash Learning from Human Feedback1 Dec
  9. 2023Is RLHF More Difficult than Standard RL?25 Jun
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  7. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  9. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  10. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 9
  11. 2025No-Regret Generative Modeling via Parabolic Monge-Amp`ere PDEcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  2. 2026Groupwise Distortion Guarantees for Preference-Based Alignment
  3. 2025Influencing Humans to Conform to Preference Models for RLHF
  4. 2025Maximizing the efficiency of human feedback in AI alignment: a comparative analysis
  5. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  8. 2023A density estimation perspective on learning from pairwise human preferences