# Subgroups Matter for Robust Bias Mitigation

arXiv: https://arxiv.org/abs/2505.21363 · HTML: https://mlsanity.com/paper/2505.21363/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021)
- [On Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2210.11369/) (2022)
- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Conditional Learning of Fair Representations](https://mlsanity.com/paper/1910.07162/) (2019)
- [Robust Optimization for Fairness with Noisy Protected Groups](https://mlsanity.com/paper/2002.09343/) (2020)
- [MEDFAIR: Benchmarking Fairness for Medical Imaging](https://mlsanity.com/paper/2210.01725/) (2022)
- [Examining and Combating Spurious Features under Distribution Shift](https://mlsanity.com/paper/2106.07171/) (2021)
- [Estimating Generalization under Distribution Shifts via Domain-Invariant Representations](https://mlsanity.com/paper/2007.03511/) (2020)
- [Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers](https://mlsanity.com/paper/2203.04913/) (2022)
- [A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers](https://mlsanity.com/paper/2207.03277/) (2022)
- [Discovering environments with XRM](https://mlsanity.com/paper/2309.16748/) (2023)
- [UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models](https://mlsanity.com/paper/2110.15499/) (2021)
- [Mitigating Dataset Bias by Using Per-sample Gradient](https://mlsanity.com/paper/2205.15704/) (2022)
- [Causal Balancing for Domain Generalization](https://mlsanity.com/paper/2206.05263/) (2022)
- [Coarse race data conceals disparities in clinical risk score performance](https://mlsanity.com/paper/2304.09270/) (2023)
- [On the use of Mahalanobis distance for out-of-distribution detection with neural networks for medical imaging](https://mlsanity.com/paper/2309.01488/) (2023)
- [Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference](https://mlsanity.com/paper/2404.13815/) (2024)
- [Distributionally Robust Optimization with Probabilistic Group](https://mlsanity.com/paper/2303.05809/) (2023)
- [Slicing Through Bias: Explaining Performance Gaps in Medical Image Analysis using Slice Discovery Methods](https://mlsanity.com/paper/2406.12142/) (2024)

## Built on by (most cited first, 2 of 2)

- [Representation Invariance and Allocation: When Subgroup Balance Matters](https://mlsanity.com/paper/2512.09496/) (2025)
- [Reducing the Adaptation Gap Through Reachable Fisher Geometry](https://mlsanity.com/paper/2609.36329/) (2026)

## Abstract (arXiv)

Despite the constant development of new bias mitigation methods for machine learning, no method consistently succeeds, and a fundamental question remains unanswered: when and why do bias mitigation techniques fail? In this paper, we hypothesise that a key factor may be the often-overlooked but crucial step shared by many bias mitigation methods: the definition of subgroups. To investigate this, we conduct a comprehensive evaluation of state-of-the-art bias mitigation methods across multiple vision and language classification tasks, systematically varying subgroup definitions, including coarse, fine-grained, intersectional, and noisy subgroups. Our results reveal that subgroup choice significantly impacts performance, with certain groupings paradoxically leading to worse outcomes than no mitigation at all. Our findings suggest that observing a disparity between a set of subgroups is not a sufficient reason to use those subgroups for mitigation. Through theoretical analysis, we explain these phenomena and uncover a counter-intuitive insight that, in some cases, improving fairness with respect to a particular set of subgroups is best achieved by using a different set of subgroups for mitigation. Our work highlights the importance of careful subgroup definition in bias mitigation and presents it as an alternative lever for improving the robustness and fairness of machine learning models.