mlsanityData to 14 Oct 2026

Interpretation of Black Box NLP Models: A Survey

arXiv 2203.17081 ↗First posted 31 Mar 2022cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

An increasing number of machine learning models have been deployed in domains with high stakes such as finance and healthcare. Despite their superior performances, many models are black boxes in nature which are hard to explain. There are growing efforts for researchers to develop methods to interpret these black-box models. Post hoc explanations based on perturbations, such as LIME, are widely used approaches to interpret a machine learning model after it has been built. This class of methods has been shown to exhibit large instability, posing serious challenges to the effectiveness of the method itself and harming user trust. In this paper, we propose S-LIME, which utilizes a hypothesis testing framework based on central limit theorem for determining the number of perturbation points needed to guarantee stability of the resulting explanation. Experiments on both simulated and real world data sets are provided to demonstrate the effectiveness of our method.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 85 of its 128 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

85 of 128 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  12. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
38 more
  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  2. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  3. 2019Attention is not Explanationcited by 929
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  6. 2016The Mythos of Model Interpretabilitycited by 721
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  8. 2019Designing and Interpreting Probes with Control Taskscited by 640
  9. 2019Attention is not not Explanationcited by 636
  10. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  11. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  13. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
  14. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  15. 2019Is Attention Interpretable?cited by 389
  16. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  17. 2018On the Robustness of Interpretability Methodscited by 314
  18. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  19. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  20. 2020A Survey on Neural Network Interpretabilitycited by 291
  21. 2018Consistent Individualized Feature Attribution for Tree Ensemblescited by 285
  22. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  23. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  24. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  25. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  26. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  27. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  28. 2015Visualizing and Understanding Neural Models in NLPcited by 200
  29. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 199
  30. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  31. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  32. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  33. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  34. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  35. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  36. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  37. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  38. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 3; 2025: 2; 2026: 2.

9 papers.

  1. 2023Post Hoc Explanations of Language Models Can Improve Language Modelscited by 50
  2. 2022LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Modelscited by 28
  3. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  4. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  5. 2026Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemakingcited by 4
  6. 2024The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreementcited by 3
  7. 2024Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMscited by 2
  8. 2025Which Words Matter Most in Zero-Shot Prompts?cited by 2
  9. 2026Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Mediacited by 0