# Preference Elicitation for Policy Optimization and Application to Aligning Heart Transplantation with Human Values

arXiv: https://arxiv.org/abs/2608.28620 · HTML: https://mlsanity.com/paper/2608.28620/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 18 of its 62 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 18 of 18)

- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [Active classification with comparison queries](https://mlsanity.com/paper/1704.03564/) (2017)
- [Eliciting Single-Peaked Preferences Using Comparison Queries](https://mlsanity.com/paper/1401.3449/) (2014)
- [Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions](https://mlsanity.com/paper/2503.00940/) (2025)
- [Deploying a Robust Active Preference Elicitation Algorithm on MTurk: Experiment Design, Interface, and Evaluation for COVID-19 Patient Prioritization](https://mlsanity.com/paper/2306.04061/) (2023)
- [Learning Linear Utility Functions From Pairwise Comparison Queries](https://mlsanity.com/paper/2405.02612/) (2024)
- [Position-Indexed Formulations for Kidney Exchange](https://mlsanity.com/paper/1606.01623/) (2016)
- [Robust Optimization with Decision-Dependent Information Discovery](https://mlsanity.com/paper/2004.08490/) (2020)
- [Policy Optimization for Dynamic Heart Transplant Allocation](https://mlsanity.com/paper/2512.12497/) (2025)
- [Near-Optimal Dynamic Matching via Coarsening with Application to Heart Transplantation](https://mlsanity.com/paper/2602.04989/) (2026)
- [Position: Machine Learning for Heart Transplant Allocation Policy Optimization Should Account for Incentives](https://mlsanity.com/paper/2602.04990/) (2026)
- [Learning Potentials for Dynamic Matching and Application to Heart Transplantation](https://mlsanity.com/paper/2602.08878/) (2026)
- [Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Values](https://mlsanity.com/paper/2506.00079/) (2025)
- [Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis](https://mlsanity.com/paper/2606.02671/) (2026)
- [Combinatorial Optimization using Comparison Oracles](https://mlsanity.com/paper/2511.15142/) (2025)

## Built on by (most cited first, 1 of 1)

- [Efficient Learning of Survival Analysis Models Under Outcome-Dependent Truncation and Censoring](https://mlsanity.com/paper/2610.12538/) (2026)

## Abstract (arXiv)

Preference elicitation is essential for aligning AI systems with human values. Prior approaches (e.g., for organ allocation) often ask stakeholders to compare the decisions of an algorithm (e.g., patient A vs. patient B). Such a decision-level approach conflates the means with the ends. Instead, we elicit preferences directly over allocation outcomes to learn a utility function for policy optimization. We construct a novel preference elicitation algorithm for linear utilities that outperforms prior techniques in practice. Our algorithm has two phases. The first phase learns cutting planes through pairwise comparisons to rapidly shrink the space of possible attribute weights and warm-starts the second phase by eliminating dominated regions. The second phase then provably converges to the user's utility function. We apply our technique to heart transplant allocation, where a policy must balance competing objectives such as post-transplant outcomes, waitlist mortality, geographic ease, and equity. Using our algorithm, we conduct a user study to learn and aggregate a human-value-aligned utility function. We evaluate the linear representation against LLM-based methods supplied with users' natural language explanations, finding that none of these methods is more accurate than the learned linear utilities. We use the resulting utility function to optimize heart transplant policies that are significantly better aligned with human values. Compared to the hindsight optimum, the status quo policy achieves a competitive ratio of just 0.66, while our method is near-optimal with a competitive ratio of 0.95.