# Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction

arXiv: https://arxiv.org/abs/2007.12248 · HTML: https://mlsanity.com/paper/2007.12248/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2020.

## Built on by (most cited first, 30 of 36)

- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [The effectiveness of feature attribution methods and its correlation with automatic evaluation scores](https://mlsanity.com/paper/2105.14944/) (2021)
- [Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoning](https://mlsanity.com/paper/2202.08816/) (2022)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [Do Input Gradients Highlight Discriminative Features?](https://mlsanity.com/paper/2102.12781/) (2021)
- [Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks](https://mlsanity.com/paper/2107.11400/) (2021)
- [Visual correspondence-based explanations improve AI robustness and human-AI team accuracy](https://mlsanity.com/paper/2208.00780/) (2022)
- [Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales](https://mlsanity.com/paper/2305.07095/) (2023)
- [Improving Human-AI Collaboration With Descriptions of AI Behavior](https://mlsanity.com/paper/2301.06937/) (2023)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [Teaching Humans When To Defer to a Classifier via Exemplars](https://mlsanity.com/paper/2111.11297/) (2021)
- [A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making](https://mlsanity.com/paper/2205.05126/) (2022)
- [Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations](https://mlsanity.com/paper/2112.09669/) (2021)
- [On the Effect of Information Asymmetry in Human-AI Teams](https://mlsanity.com/paper/2205.01467/) (2022)
- [Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization](https://mlsanity.com/paper/2306.09299/) (2023)
- [On the Relationship Between Explanation and Prediction: A Causal View](https://mlsanity.com/paper/2212.06925/) (2022)
- [Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making](https://mlsanity.com/paper/2401.07058/) (2024)
- [Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset](https://mlsanity.com/paper/2204.11642/) (2022)
- [Monitoring Shortcut Learning using Mutual Information](https://mlsanity.com/paper/2206.13034/) (2022)
- [How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps](https://mlsanity.com/paper/2404.16042/) (2024)
- [Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting](https://mlsanity.com/paper/2410.12284/) (2024)
- [Exploring The Role of Local and Global Explanations in Recommender Systems](https://mlsanity.com/paper/2109.13301/) (2021)
- [On the Relationship Between Explanations, Fairness Perceptions, and Decisions](https://mlsanity.com/paper/2204.13156/) (2022)
- [Explaining the Road Not Taken](https://mlsanity.com/paper/2103.14973/) (2021)
- [MLOps Spanning Whole Machine Learning Life Cycle: A Survey](https://mlsanity.com/paper/2304.07296/) (2023)
- [Evaluating the Utility of Model Explanations for Model Development](https://mlsanity.com/paper/2312.06032/) (2023)
- [Intelligent Decision Assistance Versus Automated Decision-Making: Enhancing Knowledge Work Through Explainable Artificial Intelligence](https://mlsanity.com/paper/2109.13827/) (2021)
- [Responsibility: An Example-based Explainable AI approach via Training Process Inspection](https://mlsanity.com/paper/2209.03433/) (2022)
- [Hidden Conflicts in Neural Networks and Their Implications for Explainability](https://mlsanity.com/paper/2310.20363/) (2023)
- [Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments](https://mlsanity.com/paper/2105.02825/) (2021)

## Abstract (arXiv)

We present a randomized controlled trial for a model-in-the-loop regression task, with the goal of measuring the extent to which (1) good explanations of model predictions increase human accuracy, and (2) faulty explanations decrease human trust in the model. We study explanations based on visual saliency in an image-based age prediction task for which humans and learned models are individually capable but not highly proficient and frequently disagree. Our experimental design separates model quality from explanation quality, and makes it possible to compare treatments involving a variety of explanations of varying levels of quality. We find that presenting model predictions improves human accuracy. However, visual explanations of various kinds fail to significantly alter human accuracy or trust in the model - regardless of whether explanations characterize an accurate model, an inaccurate one, or are generated randomly and independently of the input image. These findings suggest the need for greater evaluation of explanations in downstream decision making tasks, better design-based tools for presenting explanations to users, and better approaches for generating explanations.