# LLM Sensitivity Evaluation Framework for Clinical Diagnosis

arXiv: https://arxiv.org/abs/2504.13475 · HTML: https://mlsanity.com/paper/2504.13475/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4](https://mlsanity.com/paper/2312.16171/) (2023)
- [Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting](https://mlsanity.com/paper/2305.12723/) (2023)

## Built on by (most cited first, 7 of 7)

- [Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving](https://mlsanity.com/paper/2506.08349/) (2025)
- [Generalist Foundation Models Are Not Clinical Enough for Hospital Operations](https://mlsanity.com/paper/2511.13703/) (2025)
- [Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs](https://mlsanity.com/paper/2605.30646/) (2026)
- [When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations](https://mlsanity.com/paper/2606.07237/) (2026)
- [MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models](https://mlsanity.com/paper/2606.24155/) (2026)
- [ShriNep@EEUCA 2026: RAKSHAK - Multi-Task DeBERTa with Rationale Distillation and Jigsaw-Augmented Training for Toxic Intent Classification](https://mlsanity.com/paper/2607.20450/) (2026)
- [Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts](https://mlsanity.com/paper/2609.38600/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive performance across various domains. However, for clinical diagnosis, higher expectations are required for LLM's reliability and sensitivity: thinking like physicians and remaining sensitive to key medical information that affects diagnostic reasoning, as subtle variations can lead to different diagnosis results. Yet, existing works focus mainly on investigating the sensitivity of LLMs to irrelevant context and overlook the importance of key information. In this paper, we investigate the sensitivity of LLMs, i.e. GPT-3.5, GPT-4, Gemini, Claude3 and LLaMA2-7b, to key medical information by introducing different perturbation strategies. The evaluation results highlight the limitations of current LLMs in remaining sensitive to key medical information for diagnostic decision-making. The evolution of LLMs must focus on improving their reliability, enhancing their ability to be sensitive to key information, and effectively utilizing this information. These improvements will enhance human trust in LLMs and facilitate their practical application in real-world scenarios. Our code and dataset are available at https://github.com/chenwei23333/DiagnosisQA.