# The Promise and Peril of Human Evaluation for Model Interpretability

arXiv: https://arxiv.org/abs/1711.07414 · HTML: https://mlsanity.com/paper/1711.07414/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2017.

## Built on by (most cited first, 30 of 52)

- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations](https://mlsanity.com/paper/2307.08678/) (2023)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning](https://mlsanity.com/paper/2401.13986/) (2024)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [Properties and Challenges of LLM-Generated Explanations](https://mlsanity.com/paper/2402.10532/) (2024)
- [A Critical Survey on Fairness Benefits of Explainable AI](https://mlsanity.com/paper/2310.13007/) (2023)
- [Counterfactual Evaluation for Explainable AI](https://mlsanity.com/paper/2109.01962/) (2021)
- [A Fine-grained Interpretability Evaluation Benchmark for Neural NLP](https://mlsanity.com/paper/2205.11097/) (2022)
- [Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence](https://mlsanity.com/paper/2112.14466/) (2021)
- [Improving Interpretation Faithfulness for Vision Transformers](https://mlsanity.com/paper/2311.17983/) (2023)
- [Towards Visual Saliency Explanations of Face Verification](https://mlsanity.com/paper/2305.08546/) (2023)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated](https://mlsanity.com/paper/2107.02033/) (2021)
- [Mediators: Conversational Agents Explaining NLP Model Behavior](https://mlsanity.com/paper/2206.06029/) (2022)
- [FLARE: Faithful Logic-Aided Reasoning and Exploration](https://mlsanity.com/paper/2410.11900/) (2024)
- [An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classification](https://mlsanity.com/paper/2209.10876/) (2022)
- [What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks](https://mlsanity.com/paper/2403.12730/) (2024)
- [Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers](https://mlsanity.com/paper/2103.14465/) (2021)
- [DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation](https://mlsanity.com/paper/2108.13140/) (2021)
- [Do explanations generalize across large reasoning models?](https://mlsanity.com/paper/2601.11517/) (2026)
- [Explainable Artificial Intelligence: Precepts, Methods, and Opportunities for Research in Construction](https://mlsanity.com/paper/2211.06579/) (2022)
- [Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination](https://mlsanity.com/paper/2410.17477/) (2024)
- [A Framework for Evaluating Post Hoc Feature-Additive Explainers](https://mlsanity.com/paper/2106.08376/) (2021)

## Abstract (arXiv)

Transparency, user trust, and human comprehension are popular ethical motivations for interpretable machine learning. In support of these goals, researchers evaluate model explanation performance using humans and real world applications. This alone presents a challenge in many areas of artificial intelligence. In this position paper, we propose a distinction between descriptive and persuasive explanations. We discuss reasoning suggesting that functional interpretability may be correlated with cognitive function and user preferences. If this is indeed the case, evaluation and optimization using functional metrics could perpetuate implicit cognitive bias in explanations that threaten transparency. Finally, we propose two potential research directions to disambiguate cognitive function and explanation models, retaining control over the tradeoff between accuracy and interpretability.