mlsanityData to 14 Oct 2026

Jackpot! Alignment as a Maximal Lottery

arXiv 2501.19266 ↗First posted 31 Jan 2025cs.AI cs.LG econ.THcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF), the standard for aligning Large Language Models (LLMs) with human values, is known to fail to satisfy properties that are intuitively desirable, such as respecting the preferences of the majority citege2024axioms. To overcome these issues, we propose the use of a probabilistic Social Choice rule called maximal lotteries as a replacement for RLHF. We show that a family of alignment techniques, namely Nash Learning from Human Feedback (NLHF) citemunos2023nash and variants, approximate maximal lottery outcomes and thus inherit its beneficial properties. We confirm experimentally that our proposed methodology handles situations that arise when working with preferences more robustly than standard RLHF, including supporting the preferences of the majority, providing principled ways of handling non-transitivities in the preference data, and robustness to irrelevant alternatives. This results in systems that better incorporate human values and respect human intentions.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 55 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

17 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Axioms for AI Alignment from Human Feedback23 May
  2. 2024Mapping Social Choice Theory to RLHF19 Apr
  3. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  4. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  5. 2023Nash Learning from Human Feedback1 Dec

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  7. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  8. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  9. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  10. 2024A theory of appropriateness with applications to generative artificial intelligencecited by 25
  11. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23
  12. 2023RLHF and IIA: Perverse Incentivescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 10.

17 papers.

Sort:

By area:

  1. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
  2. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 23
  3. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 22
  4. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matchingcited by 10
  5. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 7
  6. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 4
  7. 2026Differential Voting: Loss Functions For Axiomatically Diverse Aggregation of Heterogeneous Preferencescited by 2
  8. 2026AI Alignment From Social Choice Perspectivescited by 2
  9. 2025Adaptive Preference Aggregationcited by 1
  10. 2026The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisionscited by 1
  11. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scalingcited by 0
  12. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
5 more
  1. 2026Fair Agents: Balancing Multistakeholder Alignment in Multi-Agent Personalization Systemscited by 0
  2. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  3. 2026Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policiescited by 0
  4. 2026Inference-Time Nash Alignmentcited by 0
  5. 2026Social Choice Foundations for Simulation-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
  2. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  3. 2026Groupwise Distortion Guarantees for Preference-Based Alignment
  4. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
  5. 2025Maximizing the efficiency of human feedback in AI alignment: a comparative analysis
  6. 2026A Markov Chain Approach to Preference Alignment
  7. 2024Language Alignment via Nash-learning and Adaptive feedback
  8. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment