mlsanityData to 14 Oct 2026

ExSum: From Local Explanations to Model Understanding

arXiv 2205.00130 ↗First posted 30 Apr 2022cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Interpretability methods are developed to understand the working mechanisms of black-box models, which is crucial to their responsible deployment. Fulfilling this goal requires both that the explanations generated by these methods are correct and that people can easily and reliably understand them. While the former has been addressed in prior work, the latter is often overlooked, resulting in informal model understanding derived from a handful of local explanations. In this paper, we introduce explanation summary (ExSum), a mathematical framework for quantifying model understanding, and propose metrics for its quality assessment. On two domains, ExSum highlights various limitations in the current practice, helps develop accurate model understanding, and reveals easily overlooked properties of the model. We also connect understandability to other properties of explanations such as human alignment, robustness, and counterfactual minimality and plausibility.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 30 Apr 2022.

Builds on

27 of 36 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2018Sanity Checks for Saliency Mapscited by 1,156
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  10. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  11. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  12. 2017Interpretation of Neural Networks is Fragilecited by 444
15 more
  1. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  2. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  3. 2018On the Robustness of Interpretability Methodscited by 314
  4. 2016Rationalizing Neural Predictionscited by 281
  5. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  6. 2015Visualizing and Understanding Neural Models in NLPcited by 200
  7. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  8. 2019Interpretable Neural Predictions with Differentiable Binary Variablescited by 126
  9. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 106
  10. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  11. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  12. 2018A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizationscited by 60
  13. 2021Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Modelscited by 39
  14. 2019Evaluating Recurrent Neural Network Explanationscited by 29
  15. 2021The Irrationality of Neural Rationale Modelscited by 16

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 4; 2024: 4.

15 papers.

Sort:

By area:

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  2. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  3. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  4. 2022Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Treescited by 14
  5. 2022The Solvability of Interpretability Evaluation Metricscited by 11
  6. 2022Analyzing Explainer Robustness via Probabilistic Lipschitzness of Prediction Functionscited by 6
  7. 2022Gaussian Process Surrogate Models for Neural Networkscited by 5
  8. 2024Exploring the Trade-off Between Model Performance and Explanation Plausibility of Text Classifiers Using Human Rationalescited by 5
  9. 2022On Interactive Explanations as Non-Monotonic Reasoningcited by 3
  10. 2023Evaluating the Utility of Model Explanations for Model Developmentcited by 3
  11. 2024Explanation sensitivity to the randomness of large language models: the case of journalistic text classificationcited by 3
  12. 2023Towards Reconciling Usability and Usefulness of Explainable AI Methodologiescited by 2
3 more
  1. 2023Fixing confirmation bias in feature attribution methods via semantic matchcited by 2
  2. 2024OpenHEXAI: An Open-Source Framework for Human-Centered Evaluation of Explainable Machine Learningcited by 2
  3. 2024Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanationscited by 2