# Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare

arXiv: https://arxiv.org/abs/2502.16051 · HTML: https://mlsanity.com/paper/2502.16051/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2025.

## Built on by (most cited first, 5 of 5)

- [Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases](https://mlsanity.com/paper/2503.04691/) (2025)
- [MiraMind: Benchmarking Reliable Mental Health Reasoning beyond Answer Accuracy](https://mlsanity.com/paper/2512.09636/) (2025)
- [DICE: A Framework for Dimensional and Contextual Evaluation of Language Models](https://mlsanity.com/paper/2504.10359/) (2025)
- [One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models](https://mlsanity.com/paper/2603.03291/) (2026)
- [Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing](https://mlsanity.com/paper/2601.18061/) (2026)

## Abstract (arXiv)

Current medical language model (LM) benchmarks often over-simplify the complexities of day-to-day clinical practice tasks and instead rely on evaluating LMs on multiple-choice board exam questions. In psychiatry especially, these challenges are worsened by fairness and bias issues, since models can be swayed by patient demographics even when those factors should not influence clinical decisions. Thus, we present an expert-created and annotated dataset spanning five critical domains of decision-making in mental healthcare: treatment, diagnosis, documentation, monitoring, and triage. This U.S.-centric dataset - created without any LM assistance - is designed to capture the nuanced clinical reasoning and daily ambiguities mental health practitioners encounter, reflecting the inherent complexities of care delivery that are missing from existing datasets. Almost all base questions with five answer options each have had the decision-irrelevant demographic patient information removed and replaced with variables, e.g., for age or ethnicity, and are available for male, female, or non-binary-coded patients. This design enables systematic evaluations of model performance and bias by studying how demographic factors affect decision-making. For question categories dealing with ambiguity and multiple valid answer options, we create a preference dataset with uncertainties from the expert annotations. We outline a series of intended use cases and demonstrate the usability of our dataset by evaluating sixteen off-the-shelf and six (mental) health fine-tuned LMs on category-specific task accuracy, on the fairness impact of patient demographic information on decision-making, and how consistently free-form responses deviate from human-annotated samples.