# Bias patterns in the application of LLMs for clinical decision support: A comprehensive study

arXiv: https://arxiv.org/abs/2404.15149 · HTML: https://mlsanity.com/paper/2404.15149/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [A Clarification of the Nuances in the Fairness Metrics Landscape](https://mlsanity.com/paper/2106.00467/) (2021)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [PMC-LLaMA: Towards Building Open-source Language Models for Medicine](https://mlsanity.com/paper/2304.14454/) (2023)
- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models](https://mlsanity.com/paper/2403.12025/) (2024)
- [Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding](https://mlsanity.com/paper/2304.05368/) (2023)
- [Improving Fairness in AI Models on Electronic Health Records: The Case for Federated Learning Methods](https://mlsanity.com/paper/2305.11386/) (2023)
- [Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Management](https://mlsanity.com/paper/2108.01764/) (2021)

## Built on by (most cited first, 27 of 27)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- ["You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations](https://mlsanity.com/paper/2406.12232/) (2024)
- [A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions](https://mlsanity.com/paper/2409.16430/) (2024)
- [Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models](https://mlsanity.com/paper/2408.07665/) (2024)
- [Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs](https://mlsanity.com/paper/2412.14304/) (2024)
- [Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare](https://mlsanity.com/paper/2502.13319/) (2025)
- [Implicit Bias in LLMs: A Survey](https://mlsanity.com/paper/2503.02776/) (2025)
- [CLIMB: A Benchmark of Clinical Bias in Large Language Models](https://mlsanity.com/paper/2407.05250/) (2024)
- [Aligning (Medical) LLMs for (Counterfactual) Fairness](https://mlsanity.com/paper/2408.12055/) (2024)
- [Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models](https://mlsanity.com/paper/2504.05325/) (2025)
- [Enabling Scalable Evaluation of Bias Patterns in Medical LLMs](https://mlsanity.com/paper/2410.14763/) (2024)
- [Reward Hacking Mitigation using Verifiable Composite Rewards](https://mlsanity.com/paper/2509.15557/) (2025)
- [Compared to What? Baselines and Metrics for Counterfactual Prompting](https://mlsanity.com/paper/2605.01048/) (2026)
- [MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoning](https://mlsanity.com/paper/2510.12818/) (2025)
- [A word association network methodology for evaluating implicit biases in LLMs compared to humans](https://mlsanity.com/paper/2510.24488/) (2025)
- [Can SAEs reveal and mitigate racial biases of LLMs in healthcare?](https://mlsanity.com/paper/2511.00177/) (2025)
- [Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI](https://mlsanity.com/paper/2505.10472/) (2025)
- ['Rich Dad, Poor Lad': How do Large Language Models Contextualize Socioeconomic Factors in College Admission ?](https://mlsanity.com/paper/2509.16400/) (2025)
- [When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Making](https://mlsanity.com/paper/2603.18530/) (2026)
- [Tipta uzmanlik sinavinda (tus) buyuk dil modelleri insanlardan daha mi basarili?](https://mlsanity.com/paper/2408.12305/) (2024)
- [Evaluating the Bias in LLMs for Surveying Opinion and Decision Making in Healthcare](https://mlsanity.com/paper/2504.08260/) (2025)
- [The Value Sensitivity Gap: How Clinical Large Language Models Respond to Patient Preference Statements in Shared Decision-Making](https://mlsanity.com/paper/2603.00076/) (2026)
- [Analyzing LLM Reasoning to Uncover Mental Health Stigma](https://mlsanity.com/paper/2604.25053/) (2026)
- [Toward Better Assessment of LLMs' Performance in Clinical Error Detection](https://mlsanity.com/paper/2608.16643/) (2026)
- [Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent](https://mlsanity.com/paper/2609.03221/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have emerged as powerful candidates to inform clinical decision-making processes. While these models play an increasingly prominent role in shaping the digital landscape, two growing concerns emerge in healthcare applications: 1) to what extent do LLMs exhibit social bias based on patients' protected attributes (like race), and 2) how do design choices (like architecture design and prompting strategies) influence the observed biases? To answer these questions rigorously, we evaluated eight popular LLMs across three question-answering (QA) datasets using clinical vignettes (patient descriptions) standardized for bias evaluations. We employ red-teaming strategies to analyze how demographics affect LLM outputs, comparing both general-purpose and clinically-trained models. Our extensive experiments reveal various disparities (some significant) across protected groups. We also observe several counter-intuitive patterns such as larger models not being necessarily less biased and fined-tuned models on medical data not being necessarily better than the general-purpose models. Furthermore, our study demonstrates the impact of prompt design on bias patterns and shows that specific phrasing can influence bias patterns and reflection-type approaches (like Chain of Thought) can reduce biased outcomes effectively. Consistent with prior studies, we call on additional evaluations, scrutiny, and enhancement of LLMs used in clinical decision support applications.