# Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare

arXiv: https://arxiv.org/abs/2502.13319 · HTML: https://mlsanity.com/paper/2502.13319/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability.
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- Mechanistic interpretability / [Probing deception, bias and beliefs](https://mlsanity.com/mechinterp/probing-deception-bias-and-beliefs/)

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings](https://mlsanity.com/paper/2003.11515/) (2020)
- [Evaluating the Zero-shot Robustness of Instruction-tuned Language Models](https://mlsanity.com/paper/2306.11270/) (2023)
- [Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)](https://mlsanity.com/paper/2305.06299/) (2023)
- [Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation](https://mlsanity.com/paper/2401.13867/) (2024)
- [Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery](https://mlsanity.com/paper/2304.13714/) (2023)
- [Bias patterns in the application of LLMs for clinical decision support: A comprehensive study](https://mlsanity.com/paper/2404.15149/) (2024)
- [Open (Clinical) LLMs are Sensitive to Instruction Phrasings](https://mlsanity.com/paper/2407.09429/) (2024)
- [Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model](https://mlsanity.com/paper/2310.12611/) (2023)
- [Write It Like You See It: Detectable Differences in Clinical Notes By Race Lead To Differential Model Recommendations](https://mlsanity.com/paper/2205.03931/) (2022)
- [Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs](https://mlsanity.com/paper/2406.20086/) (2024)
- [Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing](https://mlsanity.com/paper/2501.14457/) (2025)

## Built on by (most cited first, 8 of 8)

- [Interpretability Can Be Actionable](https://mlsanity.com/paper/2605.11161/) (2026)
- [Can SAEs reveal and mitigate racial biases of LLMs in healthcare?](https://mlsanity.com/paper/2511.00177/) (2025)
- [Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues](https://mlsanity.com/paper/2601.12868/) (2026)
- [A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies](https://mlsanity.com/paper/2602.06015/) (2026)
- [Tracing Pharmacological Knowledge In Large Language Models](https://mlsanity.com/paper/2603.03407/) (2026)
- [Inference Time Causal Probing in LLMs](https://mlsanity.com/paper/2605.07631/) (2026)
- [Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding](https://mlsanity.com/paper/2608.07353/) (2026)
- [Xeno-Interpretability: Investigating the Alien Minds of LLMs](https://mlsanity.com/paper/2609.20408/) (2026)

## Abstract (arXiv)

We know from prior work that LLMs encode social biases, and that this manifests in clinical tasks. In this work we adopt tools from mechanistic interpretability to unveil sociodemographic representations and biases within LLMs in the context of healthcare. Specifically, we ask: Can we identify activations within LLMs that encode sociodemographic information (e.g., gender, race)? We find that gender information is highly localized in MLP layers and can be reliably manipulated at inference time via patching. Such interventions can surgically alter generated clinical vignettes for specific conditions, and also influence downstream clinical predictions which correlate with gender, e.g., patient risk of depression. We find that representation of patient race is somewhat more distributed, but can also be intervened upon, to a degree. To our knowledge, this is the first application of mechanistic interpretability methods to LLMs for healthcare.