# Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

arXiv: https://arxiv.org/abs/2609.38600 · HTML: https://mlsanity.com/paper/2609.38600/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 67 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities](https://mlsanity.com/paper/2301.11100/) (2023)
- [Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis](https://mlsanity.com/paper/2402.01730/) (2024)
- [An investigation into the impact of deep learning model choice on sex and race bias in cardiac MR segmentation](https://mlsanity.com/paper/2308.13415/) (2023)
- [LLM Sensitivity Evaluation Framework for Clinical Diagnosis](https://mlsanity.com/paper/2504.13475/) (2025)
- [How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study](https://mlsanity.com/paper/2504.17083/) (2025)
- [RadOnc-GPT: An Autonomous LLM Agent for Real-Time Patient Outcomes Labeling at Scale](https://mlsanity.com/paper/2509.25540/) (2025)

## Abstract (arXiv)

As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.