mlsanityData to 7 Oct 2026

Understanding Disparities in Post Hoc Machine Learning Explanation

arXiv 2401.14539 ↗First posted 25 Jan 2024cs.LG cs.CY stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Previous work has highlighted that existing post-hoc explanation methods exhibit disparities in explanation fidelity (across 'race' and 'gender' as sensitive attributes), and while a large body of work focuses on mitigating these issues at the explanation metric level, the role of the data generating process and black box model in relation to explanation disparities remains largely unexplored. Accordingly, through both simulations as well as experiments on a real-world dataset, we specifically assess challenges to explanation disparities that originate from properties of the data: limited sample size, covariate shift, concept shift, omitted variable bias, and challenges based on model properties: inclusion of the sensitive attribute and appropriate functional form. Through controlled simulation analyses, our study demonstrates that increased covariate shift, concept shift, and omission of covariates increase explanation disparities, with the effect pronounced higher for neural network models that are better able to capture the underlying functional form in comparison to linear models. We also observe consistent findings regarding the effect of concept shift and omitted variable bias on explanation disparities in the Adult income dataset. Overall, results indicate that disparities in model explanations can also depend on data and model properties. Based on this systematic investigation, we provide recommendations for the design of explanation methods that mitigate undesirable disparities.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 72 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

37 of 72 references.

References · 37

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2018Sanity Checks for Saliency Mapscited by 1,149
  6. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  7. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  8. 2018Model Cards for Model Reportingcited by 1,068
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  11. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  12. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
25 more
  1. 2019Model-Agnostic Counterfactual Explanations for Consequential Decisionscited by 204
  2. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  3. 2018Path-Specific Counterfactual Fairnesscited by 179
  4. 2015Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction modelcited by 140
  5. 2020Reliable Post hoc Explanations: Modeling Uncertainty in Explainabilitycited by 125
  6. 2020Ethical Machine Learning in Health Carecited by 112
  7. 2019The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanationscited by 97
  8. 2018Model Agnostic Supervised Local Explanationscited by 83
  9. 2018A Review of Challenges and Opportunities in Machine Learning for Healthcited by 67
  10. 2017Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillationcited by 62
  11. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  12. 2014Falling Rule Listscited by 58
  13. 2021S-LIME: Stabilized-LIME for Model Explanationcited by 52
  14. 2022Algorithmic Fairness Datasets: the Story so Farcited by 51
  15. 2018Residual Unfairness in Fair Machine Learning from Prejudiced Datacited by 49
  16. 2015Interpretable Classification Models for Recidivism Predictioncited by 44
  17. 2022The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanationscited by 41
  18. 2022Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanationscited by 40
  19. 2023Diagnosing Model Performance Under Distribution Shiftcited by 37
  20. 2020Explainability for fair machine learningcited by 33
  21. 2019Fairness and Missing Valuescited by 16
  22. 2020Causal Multi-Level Fairnesscited by 13
  23. 2019Automated Machine Learning in Practice: State of the Art and Recent Resultscited by 12
  24. 2021Developing a Fidelity Evaluation Approach for Interpretable Machine Learningcited by 8
  25. 2021What will it take to generate fairness-preserving explanations?cited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025Evaluating Model Explanations without Ground Truthcited by 6
  2. 2024GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanationscited by 5
  3. 2025Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methodscited by 3
  4. 2025AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systemscited by 2
  5. 2026Perspective on Bias in Biomedical AI: Preventing Downstream Healthcare Disparitiescited by 0
  6. 2026Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groupscited by 0