# Identifying Biased Subgroups in Ranking and Classification

arXiv: https://arxiv.org/abs/2108.07450 · HTML: https://mlsanity.com/paper/2108.07450/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 12 of 12)

- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [An Intersectional Definition of Fairness](https://mlsanity.com/paper/1807.08362/) (2018)
- [FA*IR: A Fair Top-k Ranking Algorithm](https://mlsanity.com/paper/1706.06368/) (2017)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [Measuring Fairness in Ranked Outputs](https://mlsanity.com/paper/1610.08559/) (2016)
- [Ranking with Fairness Constraints](https://mlsanity.com/paper/1704.06840/) (2017)
- [Automated Data Slicing for Model Validation:A Big data - AI Integration Approach](https://mlsanity.com/paper/1807.06068/) (2018)
- [Fairness in Ranking: A Survey](https://mlsanity.com/paper/2103.14000/) (2021)
- [Auditing and Achieving Intersectional Fairness in Classification Problems](https://mlsanity.com/paper/1911.01468/) (2019)
- [FairSearch: A Tool For Fairness in Ranked Search Results](https://mlsanity.com/paper/1905.13134/) (2019)

## Built on by (most cited first, 6 of 6)

- [A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution](https://mlsanity.com/paper/2410.21716/) (2024)
- [ShaRP: Explaining Rankings and Preferences with Shapley Values](https://mlsanity.com/paper/2401.16744/) (2024)
- [Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations](https://mlsanity.com/paper/2510.24884/) (2025)
- [Detecting algorithmic bias in medical-AI models using trees](https://mlsanity.com/paper/2312.02959/) (2023)
- [An Efficient Framework for Monitoring Subgroup Performance of Machine Learning Systems](https://mlsanity.com/paper/2212.08312/) (2022)
- [Detection of Groups with Biased Representation in Ranking](https://mlsanity.com/paper/2301.00719/) (2022)

## Abstract (arXiv)

When analyzing the behavior of machine learning algorithms, it is important to identify specific data subgroups for which the considered algorithm shows different performance with respect to the entire dataset. The intervention of domain experts is normally required to identify relevant attributes that define these subgroups. We introduce the notion of divergence to measure this performance difference and we exploit it in the context of (i) classification models and (ii) ranking applications to automatically detect data subgroups showing a significant deviation in their behavior. Furthermore, we quantify the contribution of all attributes in the data subgroup to the divergent behavior by means of Shapley values, thus allowing the identification of the most impacting attributes.