mlsanityData to 7 Oct 2026

Post-hoc Interpretability for Neural NLP: A Survey

arXiv 2108.04840 ↗First posted 10 Aug 2021cs.CL cs.LG cs.NEcited by 155

Not on the map. Why?

Abstract (arXiv)

Neural networks for NLP are becoming increasingly complex and widespread, and there is a growing concern if these models are responsible to use. Explaining models helps to address the safety and ethical concerns and is essential for accountability. Interpretability serves to provide these explanations in terms that are understandable to humans. Additionally, post-hoc methods provide explanations after a model is learned and are generally model-agnostic. This survey provides a categorization of how recent post-hoc interpretability methods communicate explanations to humans, it discusses each method in-depth, and how they are validated, as the latter is often a common concern.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 89 of its 151 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

89 of 151 references.

References · 89

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  5. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2019Attention is not Explanationcited by 916
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  13. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  14. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  15. 2019Are Sixteen Heads Really Better than One?cited by 803
  16. 2020Quantifying Attention Flow in Transformerscited by 759
  17. 2016The Mythos of Model Interpretabilitycited by 716
  18. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  19. 2019Attention is not not Explanationcited by 629
  20. 2019Designing and Interpreting Probes with Control Taskscited by 608
  21. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  22. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  23. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  24. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  25. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  26. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  27. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  28. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  29. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  30. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  31. 2019Is Attention Interpretable?cited by 387
  32. 2015Neural Module Networkscited by 375
  33. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  34. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  35. 2017The (Un)reliability of saliency methodscited by 331
  36. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  37. 2016Rationalizing Neural Predictionscited by 281
  38. 2013Direct and Indirect Effectscited by 276

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 20; 2023: 40; 2024: 50; 2025: 27; 2026: 17.

155 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  4. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  5. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  6. 2024Are self-explanations from Large Language Models faithful?cited by 101
  7. 2022Interpreting Language Models with Contrastive Explanationscited by 91
  8. 2022Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoningcited by 78
  9. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  10. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  11. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  12. 2022Interactive Model Cards: A Human-Centered Approach to Model Documentationcited by 61
38 more
  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2023Post Hoc Explanations of Language Models Can Improve Language Modelscited by 50
  3. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  4. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  5. 2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
  6. 2024Trust in AI: Progress, Challenges, and Future Directionscited by 35
  7. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  8. 2023Interpreting Pretrained Language Models via Concept Bottleneckscited by 33
  9. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  10. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 28
  11. 2022ferret: a Framework for Benchmarking Explainers on Transformerscited by 26
  12. 2022Human Interpretation of Saliency-based Explanation Over Textcited by 25
  13. 2022Scalable Interpretability via Polynomialscited by 25
  14. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  15. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  16. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  17. 2023Rationalization for Explainable NLP: A Surveycited by 22
  18. 2023Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneckcited by 21
  19. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  20. 2023Tensor networks for interpretable and efficient quantum-inspired machine learningcited by 16
  21. 2024LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanationscited by 16
  22. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  23. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  24. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  25. 2022Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methodscited by 13
  26. 2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networkscited by 13
  27. 2024Latent Concept-based Explanation of NLP Modelscited by 13
  28. 2024Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictionscited by 13
  29. 2023InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanationscited by 11
  30. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  31. 2023Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognitioncited by 10
  32. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  33. 2024Interpretability Needs a New Paradigmcited by 10
  34. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
  35. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  36. 2022Interpretation of Black Box NLP Models: A Surveycited by 9
  37. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  38. 2023Probing LLMs for Joint Encoding of Linguistic Categoriescited by 9

50 most cited of 155 All 155 (JSON)