# Adaptive Preference Aggregation

arXiv: https://arxiv.org/abs/2503.10215 · HTML: https://mlsanity.com/paper/2503.10215/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [A Minimaximalist Approach to Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2401.04056/) (2024)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Contextual Dueling Bandits](https://mlsanity.com/paper/1502.06362/) (2015)
- [Better Rewards Yield Better Summaries: Learning to Summarise Without References](https://mlsanity.com/paper/1909.01214/) (2019)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [Axioms for AI Alignment from Human Feedback](https://mlsanity.com/paper/2405.14758/) (2024)
- [α-Rank: Multi-Agent Evaluation by Evolution](https://mlsanity.com/paper/1903.01373/) (2019)
- [Re-evaluating Evaluation](https://mlsanity.com/paper/1806.02643/) (2018)
- [JuMP 1.0: Recent improvements to a modeling language for mathematical optimization](https://mlsanity.com/paper/2206.03866/) (2022)
- [Mapping Social Choice Theory to RLHF](https://mlsanity.com/paper/2404.13038/) (2024)
- [Jackpot! Alignment as a Maximal Lottery](https://mlsanity.com/paper/2501.19266/) (2025)
- [Modeling Recommender Ecosystems: Research Challenges at the Intersection of Mechanism Design, Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2309.06375/) (2023)

## Built on by (most cited first, 1 of 1)

- [Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning](https://mlsanity.com/paper/2602.19041/) (2026)

## Abstract (arXiv)

AI alignment, the challenge of ensuring AI systems act in accordance with human values, has emerged as a critical problem in the development of systems such as foundation models and recommender systems. Still, the current dominant approach, reinforcement learning with human feedback (RLHF) faces known theoretical limitations in aggregating diverse human preferences. Social choice theory provides a framework to aggregate preferences, but was not developed for the multidimensional applications typical of AI. Leveraging insights from a recently published urn process, this work introduces a preference aggregation strategy that adapts to the user's context and that inherits the good properties of the maximal lottery, a Condorcet-consistent solution concept.