Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models
Not on the map. Why?
Abstract (arXiv)
The field of eXplainable artificial intelligence (XAI) has produced a plethora of methods (e.g., saliency-maps) to gain insight into artificial intelligence (AI) models, and has exploded with the rise of deep learning (DL). However, human-participant studies question the efficacy of these methods, particularly when the AI output is wrong. In this study, we collected and analyzed 156 human-generated text and saliency-based explanations collected in a question-answering task (N=40) and compared them empirically to state-of-the-art XAI explanations (integrated gradients, conservative LRP, and ChatGPT) in a human-participant study (N=136). Our findings show that participants found human saliency maps to be more helpful in explaining AI answers than machine saliency maps, but performance negatively correlated with trust in the AI model and explanations. This finding hints at the dilemma of AI errors in explanation, where helpful explanations can lead to lower task performance when they support wrong AI predictions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 90 references that are in the map. First posted on arXiv 11 Apr 2024.
Builds on
48 of 90 references.
References · 48
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
36 more
- 2018Sanity Checks for Saliency Mapscited by 1,156
- 2017SmoothGrad: removing noise by adding noisecited by 1,109
- 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
- 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
- 2019Attention is not Explanationcited by 929
- 2016The Mythos of Model Interpretabilitycited by 721
- 2019Attention is not not Explanationcited by 636
- 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 614
- 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
- 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
- 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
- 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
- 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
- 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
- 2018Manipulating and Measuring Model Interpretabilitycited by 256
- 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
- 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 139
- 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
- 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
- 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
- 2017Explaining Recurrent Neural Network Predictions in Sentiment Analysiscited by 88
- 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
- 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
- 2019An Evaluation of the Human-Interpretability of Explanationcited by 71
- 2019Attention Interpretability Across NLP Taskscited by 71
- 2020One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparencycited by 64
- 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
- 2018What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Playcited by 53
- 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
- 2019Quantifying Interpretability and Trust in Machine Learning Systemscited by 49
- 2018The Challenge of Crafting Intelligible Intelligencecited by 47
- 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
- 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
- 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
- 2016Top-down Visual Saliency Guided by Captionscited by 22
- 2021From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidencecited by 17
Built on by
Citing papers per year: 2025: 2.
2 papers.