# Human-grounded Evaluations of Explanation Methods for Text Classification

arXiv: https://arxiv.org/abs/1908.11355 · HTML: https://mlsanity.com/paper/1908.11355/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2019.

## Built on by (most cited first, 28 of 28)

- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)
- [SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers](https://mlsanity.com/paper/2103.12279/) (2021)
- [Explainable AI: current status and future directions](https://mlsanity.com/paper/2107.07045/) (2021)
- [Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining](https://mlsanity.com/paper/2110.08412/) (2021)
- [Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations](https://mlsanity.com/paper/2112.09669/) (2021)
- [ferret: a Framework for Benchmarking Explainers on Transformers](https://mlsanity.com/paper/2208.01575/) (2022)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Learning to Scaffold: Optimizing Model Explanations for Teaching](https://mlsanity.com/paper/2204.10810/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [On the Interaction of Belief Bias and Explanations](https://mlsanity.com/paper/2106.15355/) (2021)
- [Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools](https://mlsanity.com/paper/2108.13961/) (2021)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Are Shortest Rationales the Best Explanations for Human Understanding?](https://mlsanity.com/paper/2203.08788/) (2022)
- [An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classification](https://mlsanity.com/paper/2209.10876/) (2022)
- [XPROAX-Local explanations for text classification with progressive neighborhood approximation](https://mlsanity.com/paper/2109.15004/) (2021)
- [SIDU-TXT: An XAI Algorithm for NLP with a Holistic Assessment Approach](https://mlsanity.com/paper/2402.03043/) (2024)
- [Towards Human-Centred Explainability Benchmarks For Text Classification](https://mlsanity.com/paper/2211.05452/) (2022)
- [A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning](https://mlsanity.com/paper/2406.00116/) (2024)
- [Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms](https://mlsanity.com/paper/2108.09394/) (2021)
- [Combining Transformers with Natural Language Explanations](https://mlsanity.com/paper/2110.00125/) (2021)
- [Argumentative Explanations for Pattern-Based Text Classifiers](https://mlsanity.com/paper/2205.10932/) (2022)
- [Explaining text classifiers through progressive neighborhood approximation with realistic samples](https://mlsanity.com/paper/2302.07733/) (2023)
- [Can LLM Assist in the Evaluation of the Quality of Machine Learning Explanations?](https://mlsanity.com/paper/2502.20635/) (2025)
- [A Symmetric Loss Perspective of Reliable Machine Learning](https://mlsanity.com/paper/2101.01366/) (2021)
- [Interpretable Feature Learning Framework for Smoking Behavior Detection](https://mlsanity.com/paper/2112.08178/) (2021)
- [Explainable Authorship Identification in Cultural Heritage Applications: Analysis of a New Perspective](https://mlsanity.com/paper/2311.02237/) (2023)
- [A Multi-Task Text Classification Pipeline with Natural Language Explanations: A User-Centric Evaluation in Sentiment Analysis and Offensive Language Identification in Greek Tweets](https://mlsanity.com/paper/2410.10290/) (2024)

## Abstract (arXiv)

Due to the black-box nature of deep learning models, methods for explaining the models' results are crucial to gain trust from humans and support collaboration between AIs and humans. In this paper, we consider several model-agnostic and model-specific explanation methods for CNNs for text classification and conduct three human-grounded evaluations, focusing on different purposes of explanations: (1) revealing model behavior, (2) justifying model predictions, and (3) helping humans investigate uncertain predictions. The results highlight dissimilar qualities of the various explanation methods we consider and show the degree to which these methods could serve for each purpose.