# Evaluating Large Language Models for Fair and Reliable Organ Allocation

arXiv: https://arxiv.org/abs/2504.03716 · HTML: https://mlsanity.com/paper/2504.03716/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 29 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Can Machines Learn Morality? The Delphi Experiment](https://mlsanity.com/paper/2110.07574/) (2021)
- [Measuring Fairness in Ranked Outputs](https://mlsanity.com/paper/1610.08559/) (2016)
- [Fairness in Rankings and Recommendations: An Overview](https://mlsanity.com/paper/2104.05994/) (2021)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024)
- [Stereotype or Personalization? User Identity Biases Chatbot Recommendations](https://mlsanity.com/paper/2410.05613/) (2024)
- [The Impossibility of Fair LLMs](https://mlsanity.com/paper/2406.03198/) (2024)
- [Fairness of Exposure in Light of Incomplete Exposure Estimation](https://mlsanity.com/paper/2205.12901/) (2022)
- [Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions](https://mlsanity.com/paper/2503.00940/) (2025)
- [A large language model-based approach to quantifying the effects of social determinants in liver transplant decisions](https://mlsanity.com/paper/2412.07924/) (2024)
- [FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations](https://mlsanity.com/paper/2404.06619/) (2024)

## Built on by (most cited first, 1 of 1)

- [Street-Level AI: Are Large Language Models Ready for Real-World Judgments?](https://mlsanity.com/paper/2508.08193/) (2025)

## Abstract (arXiv)

Medical institutions are considering the use of LLMs in high-stakes clinical decision-making, such as organ allocation. In such sensitive use cases, evaluating fairness is imperative. However, existing evaluation methods often fall short; benchmarks are too simplistic to capture real-world complexity, and accuracy-based metrics fail to address the absence of a clear ground truth. To realistically and fairly model organ allocation, specifically kidney allocation, we begin by testing the medical knowledge of LLMs to determine whether they understand the clinical factors required to make sound allocation decisions. Building on this foundation, we design two tasks: (1) Choose-One and (2) Rank-All. In Choose-One, LLMs select a single candidate from a list of potential candidates to receive a kidney. In this scenario, we assess fairness across demographics using traditional fairness metrics, such as proportional parity. In Rank-All, LLMs rank all candidates waiting for a kidney, reflecting real-world allocation processes more closely, where an organ is passed down a ranked list until allocated. Our evaluation on three LLMs reveals a divergence between fairness metrics: while exposure-based metrics suggest equitable outcomes, probability-based metrics uncover systematic preferential sorting, where specific groups were clustered in upper-ranking tiers. Furthermore, we observe that demographic preferences are highly task-dependent, showing inverted trends between Choose-One and Rank-All tasks, even when considering the topmost rank. Overall, our results indicate that current LLMs can introduce inequalities in real-world allocation scenarios, underscoring the urgent need for rigorous fairness evaluation and human oversight before their use in high-stakes decision-making.