# Jackpot! Alignment as a Maximal Lottery

arXiv: https://arxiv.org/abs/2501.19266 · HTML: https://mlsanity.com/paper/2501.19266/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Nash Learning from Human Feedback](https://mlsanity.com/paper/2312.00886/) (2023)
- [A Minimaximalist Approach to Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2401.04056/) (2024)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Human Alignment of Large Language Models through Online Preference Optimisation](https://mlsanity.com/paper/2403.08635/) (2024)
- [Axioms for AI Alignment from Human Feedback](https://mlsanity.com/paper/2405.14758/) (2024)
- [Preference Learning Algorithms Do Not Learn Preference Rankings](https://mlsanity.com/paper/2405.19534/) (2024)
- [AI Alignment and Social Choice: Fundamental Limitations and Policy Implications](https://mlsanity.com/paper/2310.16048/) (2023)
- [Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2212.03201/) (2022)
- [Mapping Social Choice Theory to RLHF](https://mlsanity.com/paper/2404.13038/) (2024)
- [A theory of appropriateness with applications to generative artificial intelligence](https://mlsanity.com/paper/2412.19010/) (2024)
- [On the Limitations of Elo: Real-World Games, are Transitive, not Additive](https://mlsanity.com/paper/2206.12301/) (2022)
- [RLHF and IIA: Perverse Incentives](https://mlsanity.com/paper/2312.01057/) (2023)

## Built on by (most cited first, 17 of 17)

- [Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset](https://mlsanity.com/paper/2507.09650/) (2025)
- [Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium](https://mlsanity.com/paper/2503.10990/) (2025)
- [Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?](https://mlsanity.com/paper/2505.23749/) (2025)
- [Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching](https://mlsanity.com/paper/2505.20627/) (2025)
- [Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory](https://mlsanity.com/paper/2506.12350/) (2025)
- [Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework](https://mlsanity.com/paper/2506.05619/) (2025)
- [Differential Voting: Loss Functions For Axiomatically Diverse Aggregation of Heterogeneous Preferences](https://mlsanity.com/paper/2601.18824/) (2026)
- [AI Alignment From Social Choice Perspectives](https://mlsanity.com/paper/2606.21550/) (2026)
- [Adaptive Preference Aggregation](https://mlsanity.com/paper/2503.10215/) (2025)
- [The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisions](https://mlsanity.com/paper/2605.12717/) (2026)
- [Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling](https://mlsanity.com/paper/2601.08777/) (2026)
- [Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning](https://mlsanity.com/paper/2602.19041/) (2026)
- [Fair Agents: Balancing Multistakeholder Alignment in Multi-Agent Personalization Systems](https://mlsanity.com/paper/2605.02379/) (2026)
- [Efficient Exploration for Iterative Nash Preference Optimization](https://mlsanity.com/paper/2606.01382/) (2026)
- [Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies](https://mlsanity.com/paper/2607.17316/) (2026)
- [Inference-Time Nash Alignment](https://mlsanity.com/paper/2609.08082/) (2026)
- [Social Choice Foundations for Simulation-Augmented Generation](https://mlsanity.com/paper/2609.38287/) (2026)

## Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF), the standard for aligning Large Language Models (LLMs) with human values, is known to fail to satisfy properties that are intuitively desirable, such as respecting the preferences of the majority citege2024axioms. To overcome these issues, we propose the use of a probabilistic Social Choice rule called maximal lotteries as a replacement for RLHF. We show that a family of alignment techniques, namely Nash Learning from Human Feedback (NLHF) citemunos2023nash and variants, approximate maximal lottery outcomes and thus inherit its beneficial properties. We confirm experimentally that our proposed methodology handles situations that arise when working with preferences more robustly than standard RLHF, including supporting the preferences of the majority, providing principled ways of handling non-transitivities in the preference data, and robustness to irrelevant alternatives. This results in systems that better incorporate human values and respect human intentions.