# MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering

arXiv: https://arxiv.org/abs/2406.06573 · HTML: https://mlsanity.com/paper/2406.06573/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Adversarially Robust Generalization Requires More Data](https://mlsanity.com/paper/1804.11285/) (2018)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)

## Built on by (most cited first, 26 of 26)

- [X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains](https://mlsanity.com/paper/2505.03981/) (2025)
- [A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcare](https://mlsanity.com/paper/2502.15871/) (2025)
- [Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning](https://mlsanity.com/paper/2502.07143/) (2025)
- [Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine](https://mlsanity.com/paper/2406.02394/) (2024)
- [Statistical inference on black-box generative models in the data kernel perspective space](https://mlsanity.com/paper/2410.01106/) (2024)
- [Medical large language models are easily distracted](https://mlsanity.com/paper/2504.01201/) (2025)
- [MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.20824/) (2025)
- [Trustworthy Medical Question Answering: An Evaluation-Centric Survey](https://mlsanity.com/paper/2506.03659/) (2025)
- [MEG: Medical Knowledge-Augmented Large Language Models for Question Answering](https://mlsanity.com/paper/2411.03883/) (2024)
- [FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2505.19562/) (2025)
- [Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation](https://mlsanity.com/paper/2602.17316/) (2026)
- [MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors](https://mlsanity.com/paper/2604.06846/) (2026)
- [Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources](https://mlsanity.com/paper/2409.11783/) (2024)
- [Are Clinical T5 Models Better for Clinical Text?](https://mlsanity.com/paper/2412.05845/) (2024)
- [Small Agent Group is the Future of Digital Health](https://mlsanity.com/paper/2602.08013/) (2026)
- [PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts](https://mlsanity.com/paper/2605.17028/) (2026)
- [CARE-ECG: Causal Agent-based Reasoning for Explainable and Counterfactual ECG Interpretation](https://mlsanity.com/paper/2604.10420/) (2026)
- [Do "New Snow Tablets" Contain Snow? Large Language Models Over-Rely on Names to Identify Ingredients of Chinese Drugs](https://mlsanity.com/paper/2504.03786/) (2025)
- [Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations](https://mlsanity.com/paper/2509.10963/) (2025)
- [NanoFlux: Adversarial Dual-LLM Evaluation and Distillation For Multi-Domain Reasoning](https://mlsanity.com/paper/2509.23252/) (2025)
- [When prompt perturbations break your A/B test: A valid statistical test for generative surveying](https://mlsanity.com/paper/2605.27463/) (2026)
- [Iterative Tree Analysis for Medical Critics](https://mlsanity.com/paper/2501.10642/) (2025)
- [Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering](https://mlsanity.com/paper/2510.18691/) (2025)
- [How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts](https://mlsanity.com/paper/2604.11133/) (2026)
- [When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations](https://mlsanity.com/paper/2606.07237/) (2026)
- [Untangling the Mechanisms of Misleading Context in Medical Question Answering](https://mlsanity.com/paper/2609.02754/) (2026)

## Abstract (arXiv)

Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance generalizes to real-world clinical settings. Medical question-answering benchmarks rely on assumptions consistent with quantifying LLM performance but that may not hold in the open world of the clinic. Yet LLMs learn broad knowledge that can help the LLM generalize to practical conditions regardless of unrealistic assumptions in celebrated benchmarks. We seek to quantify how well LLM medical question-answering benchmark performance generalizes when benchmark assumptions are violated. Specifically, we present an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz attempts to modify benchmark questions in ways aimed at confounding the LLM. We demonstrate the approach by targeting strong assumptions about patient characteristics presented in the MedQA benchmark. Successful "attacks" modify a benchmark item in ways that would be unlikely to fool a medical expert but nonetheless "trick" the LLM into changing from a correct to an incorrect answer. Further, we present a permutation test technique that can ensure a successful attack is statistically significant. We show how to use performance on a "MedFuzzed" benchmark, as well as individual successful attacks. The methods show promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.