mlsanityData to 14 Oct 2026

Can You Trust Your Preference Models? Stackelberg Games for Alignment

arXiv 2610.15008 ↗First posted 10 Oct 2026cs.AI cs.GTcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods also fits: Human values and AI feedback alignment

Abstract (arXiv)

Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent-Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle's preference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 56 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

33 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning10 Mar
  2. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment25 Feb
  3. 2024Active Preference Learning for Large Language Models12 Feb
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  10. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  11. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  12. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
17 more
  1. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  2. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  3. 2023Nash Learning from Human Feedbackcited by 224
  4. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  5. 2018Categorizing Variants of Goodhart's Lawcited by 149
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  7. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  8. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 46
  9. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 40
  10. 2022Uncertainty Estimation for Language Reward Modelscited by 37
  11. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 34
  12. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 27
  13. 2025Adversarial Training of Reward Modelscited by 18
  14. 2024Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimizationcited by 16
  15. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 10
  16. 2025Mitigating Preference Hacking in Policy Optimization with Pessimismcited by 8
  17. 2016Bayesian Reinforcement Learning: A Surveycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  2. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  3. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  4. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
  5. 2024Bandits with Preference Feedback: A Stackelberg Game Perspective
  6. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
  7. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  8. 2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees