# Are demographically invariant models and representations in medical imaging fair?

arXiv: https://arxiv.org/abs/2305.01397 · HTML: https://mlsanity.com/paper/2305.01397/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 62 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 30 of 43)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Domain Generalization using Causal Matching](https://mlsanity.com/paper/2006.07500/) (2020)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Path-Specific Counterfactual Fairness](https://mlsanity.com/paper/1802.08139/) (2018)
- [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Domain Generalization via Conditional Invariant Representation](https://mlsanity.com/paper/1807.08479/) (2018)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift](https://mlsanity.com/paper/2003.04475/) (2020)
- [Causality-inspired Single-source Domain Generalization for Medical Image Segmentation](https://mlsanity.com/paper/2111.12525/) (2021)
- [Roto-Translation Covariant Convolutional Networks for Medical Image Analysis](https://mlsanity.com/paper/1804.03393/) (2018)
- [Support and Invertibility in Domain-Invariant Representations](https://mlsanity.com/paper/1903.03448/) (2019)
- [Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests](https://mlsanity.com/paper/2106.00545/) (2021)
- [Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment](https://mlsanity.com/paper/1903.01689/) (2019)
- [FairPrune: Achieving Fairness Through Pruning for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2203.02110/) (2022)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers](https://mlsanity.com/paper/2203.04913/) (2022)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [Improving the Fairness of Chest X-ray Classifiers](https://mlsanity.com/paper/2203.12609/) (2022)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations](https://mlsanity.com/paper/2107.00520/) (2021)
- [Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities](https://mlsanity.com/paper/2102.04257/) (2021)
- [Fundamental Limits and Tradeoffs in Invariant Representation Learning](https://mlsanity.com/paper/2012.10713/) (2020)
- [Causal Conceptions of Fairness and their Consequences](https://mlsanity.com/paper/2207.05302/) (2022)
- [The Sensitivity of Counterfactual Fairness to Unmeasured Confounding](https://mlsanity.com/paper/1907.01040/) (2019)

## Built on by (most cited first, 6 of 6)

- [The Limits of Fair Medical Imaging AI In The Wild](https://mlsanity.com/paper/2312.10083/) (2023)
- [Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications](https://mlsanity.com/paper/2407.16953/) (2024)
- [Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation](https://mlsanity.com/paper/2605.06891/) (2026)
- [Causal Representation Learning with Observational Grouping for CXR Classification](https://mlsanity.com/paper/2506.20582/) (2025)
- [Causal Transfer in Medical Image Analysis](https://mlsanity.com/paper/2603.24388/) (2026)
- [Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias](https://mlsanity.com/paper/2609.32004/) (2026)

## Abstract (arXiv)

Medical imaging models have been shown to encode information about patient demographics such as age, race, and sex in their latent representation, raising concerns about their potential for discrimination. Here, we ask whether requiring models not to encode demographic attributes is desirable. We point out that marginal and class-conditional representation invariance imply the standard group fairness notions of demographic parity and equalized odds, respectively. In addition, however, they require matching the risk distributions, thus potentially equalizing away important group differences. Enforcing the traditional fairness notions directly instead does not entail these strong constraints. Moreover, representationally invariant models may still take demographic attributes into account for deriving predictions, implying unequal treatment - in fact, achieving representation invariance may require doing so. In theory, this can be prevented using counterfactual notions of (individual) fairness or invariance. We caution, however, that properly defining medical image counterfactuals with respect to demographic attributes is fraught with challenges. Finally, we posit that encoding demographic attributes may even be advantageous if it enables learning a task-specific encoding of demographic features that does not rely on social constructs such as 'race' and 'gender.' We conclude that demographically invariant representations are neither necessary nor sufficient for fairness in medical imaging. Models may need to encode demographic attributes, lending further urgency to calls for comprehensive model fairness assessments in terms of predictive performance across diverse patient groups.