# Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated

arXiv: https://arxiv.org/abs/2107.02033 · HTML: https://mlsanity.com/paper/2107.02033/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 1 Jul 2021.

## Builds on (most cited first, 11 of 11)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [The Promise and Peril of Human Evaluation for Model Interpretability](https://mlsanity.com/paper/1711.07414/) (2017)
- [Quantifying Interpretability and Trust in Machine Learning Systems](https://mlsanity.com/paper/1901.08558/) (2019)
- [Understanding and Comparing Deep Neural Networks for Age and Gender Classification](https://mlsanity.com/paper/1708.07689/) (2017)

## Built on by (most cited first, 7 of 7)

- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [Benchmarking XAI Explanations with Human-Aligned Evaluations](https://mlsanity.com/paper/2411.02470/) (2024)
- [Explainable AI needs formalization](https://mlsanity.com/paper/2409.14590/) (2024)
- [Training Characteristic Functions with Reinforcement Learning: XAI-methods play Connect Four](https://mlsanity.com/paper/2202.11797/) (2022)
- [Data augmentation and explainability for bias discovery and mitigation in deep learning](https://mlsanity.com/paper/2308.09464/) (2023)
- [Towards ML Methods for Biodiversity: A Novel Wild Bee Dataset and Evaluations of XAI Methods for ML-Assisted Rare Species Annotations](https://mlsanity.com/paper/2206.07497/) (2022)
- [Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?](https://mlsanity.com/paper/2609.35020/) (2026)

## Abstract (arXiv)

The field explainable artificial intelligence (XAI) has brought about an arsenal of methods to render Machine Learning (ML) predictions more interpretable. But how useful explanations provided by transparent ML methods are for humans remains difficult to assess. Here we investigate the quality of interpretable computer vision algorithms using techniques from psychophysics. In crowdsourced annotation tasks we study the impact of different interpretability approaches on annotation accuracy and task time. We compare these quality metrics with classical XAI, automated quality metrics. Our results demonstrate that psychophysical experiments allow for robust quality assessment of transparency in machine learning. Interestingly the quality metrics computed without humans in the loop did not provide a consistent ranking of interpretability methods nor were they representative for how useful an explanation was for humans. These findings highlight the potential of methods from classical psychophysics for modern machine learning applications. We hope that our results provide convincing arguments for evaluating interpretability in its natural habitat, human-ML interaction, if the goal is to obtain an authentic assessment of interpretability.