# Algorithmic encoding of protected characteristics in image-based models for disease detection

arXiv: https://arxiv.org/abs/2110.14755 · HTML: https://mlsanity.com/paper/2110.14755/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Censoring Representations with an Adversary](https://mlsanity.com/paper/1511.05897/) (2015)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)
- [Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning](https://mlsanity.com/paper/1708.09843/) (2017)
- [Can You Trust This Prediction? Auditing Pointwise Reliability After Learning](https://mlsanity.com/paper/1901.00403/) (2019)
- [To Split or Not to Split: The Impact of Disparate Treatment in Classification](https://mlsanity.com/paper/2002.04788/) (2020)

## Built on by (most cited first, 7 of 7)

- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [On Fairness of Medical Image Classification with Multiple Sensitive Attributes via Learning Orthogonal Representations](https://mlsanity.com/paper/2301.01481/) (2023)
- [Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models](https://mlsanity.com/paper/2304.03218/) (2023)
- [Debiased Noise Editing on Foundation Models for Fair Medical Image Classification](https://mlsanity.com/paper/2403.06104/) (2024)
- [On the Interplay of Human-AI Alignment,Fairness, and Performance Trade-offs in Medical Imaging](https://mlsanity.com/paper/2505.10231/) (2025)
- [DeepRepViz: Identifying Confounders in Deep Learning Model Predictions](https://mlsanity.com/paper/2309.15551/) (2023)
- [Thinking Outside the Box: Orthogonal Approach to Equalizing Protected Attributes](https://mlsanity.com/paper/2311.14733/) (2023)

## Abstract (arXiv)

It has been rightfully emphasized that the use of AI for clinical decision making could amplify health disparities. An algorithm may encode protected characteristics, and then use this information for making predictions due to undesirable correlations in the (historical) training data. It remains unclear how we can establish whether such information is actually used. Besides the scarcity of data from underserved populations, very little is known about how dataset biases manifest in predictive models and how this may result in disparate performance. This article aims to shed some light on these issues by exploring new methodology for subgroup analysis in image-based disease detection models. We utilize two publicly available chest X-ray datasets, CheXpert and MIMIC-CXR, to study performance disparities across race and biological sex in deep learning models. We explore test set resampling, transfer learning, multitask learning, and model inspection to assess the relationship between the encoding of protected characteristics and disease detection performance across subgroups. We confirm subgroup disparities in terms of shifted true and false positive rates which are partially removed after correcting for population and prevalence shifts in the test sets. We further find a previously used transfer learning method to be insufficient for establishing whether specific patient information is used for making predictions. The proposed combination of test-set resampling, multitask learning, and model inspection reveals valuable new insights about the way protected characteristics are encoded in the feature representations of deep neural networks.