# Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

arXiv: https://arxiv.org/abs/2505.20627 · HTML: https://mlsanity.com/paper/2505.20627/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [RLHF Workflow: From Reward Modeling to Online RLHF](https://mlsanity.com/paper/2405.07863/) (2024)
- [Self-Play Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2405.00675/) (2024)
- [Nash Learning from Human Feedback](https://mlsanity.com/paper/2312.00886/) (2023)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization](https://mlsanity.com/paper/2405.16455/) (2024)
- [Axioms for AI Alignment from Human Feedback](https://mlsanity.com/paper/2405.14758/) (2024)
- [AI Alignment and Social Choice: Fundamental Limitations and Policy Implications](https://mlsanity.com/paper/2310.16048/) (2023)
- [Mapping Social Choice Theory to RLHF](https://mlsanity.com/paper/2404.13038/) (2024)
- [An Overview of Large Language Models for Statisticians](https://mlsanity.com/paper/2502.17814/) (2025)
- [Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium](https://mlsanity.com/paper/2503.10990/) (2025)
- [Jackpot! Alignment as a Maximal Lottery](https://mlsanity.com/paper/2501.19266/) (2025)

## Built on by (most cited first, 10 of 10)

- [Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory](https://mlsanity.com/paper/2506.12350/) (2025)
- [Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework](https://mlsanity.com/paper/2506.05619/) (2025)
- [Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment](https://mlsanity.com/paper/2604.05965/) (2026)
- [Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling](https://mlsanity.com/paper/2601.08777/) (2026)
- [Efficient Exploration for Iterative Nash Preference Optimization](https://mlsanity.com/paper/2606.01382/) (2026)
- [DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination](https://mlsanity.com/paper/2606.08068/) (2026)
- [A Markov Chain Approach to Preference Alignment](https://mlsanity.com/paper/2606.22652/) (2026)
- [Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration](https://mlsanity.com/paper/2608.07419/) (2026)
- [Population-Level Generative Modeling for Ranking Data](https://mlsanity.com/paper/2608.08422/) (2026)
- [AI Alignment through a Game-theoretic Lens: A Survey](https://mlsanity.com/paper/2608.27910/) (2026)

## Abstract (arXiv)

Nash Learning from Human Feedback is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game. However, using raw preference as the payoff in the game highly limits the potential of the game-theoretic LLM alignment framework. In this paper, we systematically study using what choices of payoff based on the pairwise human preferences can yield desirable alignment properties. We establish necessary and sufficient conditions for Condorcet consistency, diversity through mixed strategies, and Smith consistency. These results provide a theoretical foundation for the robustness of game-theoretic LLM alignment. Further, we show the impossibility of preference matching -- i.e., no smooth and learnable mappings of pairwise preferences can guarantee a unique Nash equilibrium that matches a target policy, even under standard assumptions like the Bradley-Terry-Luce model. This result highlights the fundamental limitation of game-theoretic LLM alignment.