# Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models

arXiv: https://arxiv.org/abs/2412.17803 · HTML: https://mlsanity.com/paper/2412.17803/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature](https://mlsanity.com/paper/2405.04819/) (2024)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [Multimodal Machine Learning for Automated ICD Coding](https://mlsanity.com/paper/1810.13348/) (2018)
- [Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study](https://mlsanity.com/paper/2103.06511/) (2021)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [Residual-based Language Models are Free Boosters for Biomedical Imaging](https://mlsanity.com/paper/2403.17343/) (2024)
- [ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding](https://mlsanity.com/paper/2408.11363/) (2024)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models](https://mlsanity.com/paper/2409.19492/) (2024)
- [Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts](https://mlsanity.com/paper/2403.13786/) (2024)
- [Token Imbalance Adaptation for Radiology Report Generation](https://mlsanity.com/paper/2304.09185/) (2023)
- [Thought Graph: Generating Thought Process for Biological Reasoning](https://mlsanity.com/paper/2403.07144/) (2024)
- [Evaluating the Impact of Social Determinants on Health Prediction in the Intensive Care Unit](https://mlsanity.com/paper/2305.12622/) (2023)
- [Time Matters: Examine Temporal Effects on Biomedical Language Models](https://mlsanity.com/paper/2407.17638/) (2024)
- [Length-Aware Multi-Kernel Transformer for Long Document Classification](https://mlsanity.com/paper/2405.07052/) (2024)
- [Prediction of ICD Codes with Clinical BERT Embeddings and Text Augmentation with Label Balancing using MIMIC-III](https://mlsanity.com/paper/2008.10492/) (2020)

## Built on by (most cited first, 4 of 4)

- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts](https://mlsanity.com/paper/2502.08825/) (2025)
- [A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Application: Data Utility and Quality Perspectives](https://mlsanity.com/paper/2506.16594/) (2025)
- [Resource-Conscious Modeling for Next- Day Discharge Prediction Using Clinical Notes](https://mlsanity.com/paper/2604.03498/) (2026)

## Abstract (arXiv)

Data imbalance is a fundamental challenge in applying language models to biomedical applications, particularly in ICD code prediction tasks where label and demographic distributions are uneven. While state-of-the-art language models have been increasingly adopted in biomedical tasks, few studies have systematically examined how data imbalance affects model performance and fairness across demographic groups. This study fills the gap by statistically probing the relationship between data imbalance and model performance in ICD code prediction. We analyze imbalances in a standard benchmark data across gender, age, ethnicity, and social determinants of health by state-of-the-art biomedical language models. By deploying diverse performance metrics and statistical analyses, we explore the influence of data imbalance on performance variations and demographic fairness. Our study shows that data imbalance significantly impacts model performance and fairness, but feature similarity to the majority class may be a more critical factor. We believe this study provides valuable insights for developing more equitable and robust language models in healthcare applications.