mlsanityData to 7 Oct 2026

Understanding Text Classification Data and Models Using Aggregated Input Salience

arXiv 2211.05485 ↗First posted 10 Nov 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Realizing when a model is right for a wrong reason is not trivial and requires a significant effort by model developers. In some cases an input salience method, which highlights the most important parts of the input, may reveal problematic reasoning. But scrutinizing highlights over many data instances is tedious and often infeasible. Furthermore, analyzing examples in isolation does not reveal general patterns in the data or in the model's behavior. In this paper we aim to address these issues and go from understanding single examples to understanding entire datasets and models. The methodology we propose is based on aggregated salience maps, to which we apply clustering, nearest neighbor search and visualizations. Using this methodology we address multiple distinct but common model developer needs by showing how problematic data and model behavior can be identified and explained -- a necessary first step for improving the model.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

47 of 70 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2018Model Cards for Model Reportingcited by 1,068
35 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  3. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  6. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  7. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  9. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  10. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  11. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 243
  12. 2016Understanding Neural Networks through Representation Erasurecited by 223
  13. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  14. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  15. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  16. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  17. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  18. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  19. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  20. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 105
  21. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  22. 2022Interpreting Language Models with Contrastive Explanationscited by 91
  23. 2019AllenNLP Interpret: A Framework for Explaining Predictions of NLP Modelscited by 74
  24. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  25. 2021Explanation-Based Human Debugging of NLP Models: A Surveycited by 53
  26. 2014Extraction of Salient Sentences from Labelled Documentscited by 50
  27. 2020FIND: Human-in-the-Loop Debugging Deep Text Classifierscited by 40
  28. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  29. 2016Explaining Predictions of Non-Linear Classifiers in NLPcited by 37
  30. 2019From Clustering to Cluster Explanations via Neural Networkscited by 34
  31. 2022LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Modelscited by 28
  32. 2021Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updatescited by 27
  33. 2022Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Languagecited by 21
  34. 2020Exposing Shallow Heuristics of Relation Extraction Models with Challenge Datacited by 20
  35. 2022IMACS: Image Model Attribution Comparison Summariescited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18
  2. 2023Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distributioncited by 10