# Explainability's Gain is Optimality's Loss? -- How Explanations Bias Decision-making

arXiv: https://arxiv.org/abs/2206.08705 · HTML: https://mlsanity.com/paper/2206.08705/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [Efficient Search for Diverse Coherent Explanations](https://mlsanity.com/paper/1901.04909/) (2019)
- ['It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions](https://mlsanity.com/paper/1801.10408/) (2018)
- [Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactuals](https://mlsanity.com/paper/2103.11972/) (2021)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [From Data to Decisions: Distributionally Robust Optimization is Optimal](https://mlsanity.com/paper/1704.04118/) (2017)
- [The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets](https://mlsanity.com/paper/2009.11023/) (2020)

## Built on by (most cited first, 4 of 4)

- [Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat](https://mlsanity.com/paper/2307.05350/) (2023)
- [Clinicians' Voice: Fundamental Considerations for XAI in Healthcare](https://mlsanity.com/paper/2411.04855/) (2024)
- [Fixing confirmation bias in feature attribution methods via semantic match](https://mlsanity.com/paper/2307.00897/) (2023)
- [The Duet of Representations and How Explanations Exacerbate It](https://mlsanity.com/paper/2402.08379/) (2024)

## Abstract (arXiv)

Decisions in organizations are about evaluating alternatives and choosing the one that would best serve organizational goals. To the extent that the evaluation of alternatives could be formulated as a predictive task with appropriate metrics, machine learning algorithms are increasingly being used to improve the efficiency of the process. Explanations help to facilitate communication between the algorithm and the human decision-maker, making it easier for the latter to interpret and make decisions on the basis of predictions by the former. Feature-based explanations' semantics of causal models, however, induce leakage from the decision-maker's prior beliefs. Our findings from a field experiment demonstrate empirically how this leads to confirmation bias and disparate impact on the decision-maker's confidence in the predictions. Such differences can lead to sub-optimal and biased decision outcomes.