# Explanations as Bias Detectors: A Critical Study of Local Post-hoc XAI Methods for Fairness Exploration

arXiv: https://arxiv.org/abs/2505.00802 · HTML: https://mlsanity.com/paper/2505.00802/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 1 May 2025.

## Builds on (most cited first, 28 of 28)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations](https://mlsanity.com/paper/2106.00786/) (2021)
- [Contextualizing Hate Speech Classifiers with Post-hoc Explanation](https://mlsanity.com/paper/2005.02439/) (2020)
- [Fairness in Rankings and Recommendations: An Overview](https://mlsanity.com/paper/2104.05994/) (2021)
- [Explainability for fair machine learning](https://mlsanity.com/paper/2010.07389/) (2020)
- [Global Aggregations of Local Explanations for Black Box models](https://mlsanity.com/paper/1907.03039/) (2019)
- [Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey](https://mlsanity.com/paper/2309.13933/) (2023)
- [GLOBE-CE: A Translation-Based Approach for Global Counterfactual Explanations](https://mlsanity.com/paper/2305.17021/) (2023)
- [Marrying Fairness and Explainability in Supervised Learning](https://mlsanity.com/paper/2204.02947/) (2022)
- [Machine Learning Data Practices through a Data Curation Lens: An Evaluation Framework](https://mlsanity.com/paper/2405.02703/) (2024)
- [A Critical Survey on Fairness Benefits of Explainable AI](https://mlsanity.com/paper/2310.13007/) (2023)
- [LimeOut: An Ensemble Approach To Improve Process Fairness](https://mlsanity.com/paper/2006.10531/) (2020)
- [Biased Models Have Biased Explanations](https://mlsanity.com/paper/2012.10986/) (2020)
- [TABCF: Counterfactual Explanations for Tabular Data Using a Transformer-Based VAE](https://mlsanity.com/paper/2410.10463/) (2024)
- [Measuring Unfairness through Game-Theoretic Interpretability](https://mlsanity.com/paper/1910.05591/) (2019)
- [Toward Understanding the Disagreement Problem in Neural Network Feature Attribution](https://mlsanity.com/paper/2404.11330/) (2024)
- [Making ML models fairer through explanations: the case of LimeOut](https://mlsanity.com/paper/2011.00603/) (2020)

## Built on by (most cited first, 2 of 2)

- [Data-Driven Analysis of Intersectional Bias in Image Classification: A Framework with Bias-Weighted Augmentation](https://mlsanity.com/paper/2510.16072/) (2025)
- [The Impact of Machine Learning Uncertainty on the Robustness of Counterfactual Explanations](https://mlsanity.com/paper/2602.00063/) (2026)

## Abstract (arXiv)

As Artificial Intelligence (AI) is increasingly used in areas that significantly impact human lives, concerns about fairness and transparency have grown, especially regarding their impact on protected groups. Recently, the intersection of explainability and fairness has emerged as an important area to promote responsible AI systems. This paper explores how explainability methods can be leveraged to detect and interpret unfairness. We propose a pipeline that integrates local post-hoc explanation methods to derive fairness-related insights. During the pipeline design, we identify and address critical questions arising from the use of explanations as bias detectors such as the relationship between distributive and procedural fairness, the effect of removing the protected attribute, the consistency and quality of results across different explanation methods, the impact of various aggregation strategies of local explanations on group fairness evaluations, and the overall trustworthiness of explanations as bias detectors. Our results show the potential of explanation methods used for fairness while highlighting the need to carefully consider the aforementioned critical aspects.