mlsanityData to 7 Oct 2026

Explainability in Practice: A Survey of Explainable NLP Across Various Domains

arXiv 2502.00837 ↗First posted 2 Feb 2025cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing (NLP) is now embedded in critical sectors including healthcare, finance, and customer relationship management, where models such as GPT-4o, Gemini, and BERT increasingly inform decisions. The black-box nature of these models has created an urgent need for transparency. This review examines explainable NLP (XNLP) as it is actually deployed, working through seven application domains: medicine, finance, systematic reviews, customer relationship management, chatbots, social and behavioral science, and human resources. For each domain, we ask what kind of explanation the setting needs, which methods are used there, and how they are evaluated. A structured cross-domain synthesis then contrasts how those requirements diverge. We compare the main explanation method families on scope, evidence of faithfulness, and computational cost. We also propose a two-tier evaluation protocol that separates a shared technical core of metrics from the domain-specific validation layer through which those metrics have to be read. The review also addresses areas that remain underrepresented in the XNLP literature, including real-world applicability, the gap between fidelity and faithfulness, and the role of human judgment in assessing explanations. It closes with research directions, among them personalized explanations, human-in-the-loop evaluation, and mechanistic interpretability for large language models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 134 of its 285 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

134 of 285 references.

References · 134

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  8. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
38 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  3. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  10. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  11. 2019Attention is not Explanationcited by 916
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  13. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  14. 2019Attention is not not Explanationcited by 629
  15. 2019Designing and Interpreting Probes with Control Taskscited by 608
  16. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  17. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  18. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  19. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  20. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  21. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  22. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  23. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  24. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  25. 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 344
  26. 2023Explainability for Large Language Models: A Surveycited by 331
  27. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  28. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  29. 2023Faithful Chain-of-Thought Reasoningcited by 285
  30. 2016Rationalizing Neural Predictionscited by 281
  31. 2018Metrics for Explainable AI: Challenges and Prospectscited by 257
  32. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  33. 2018Techniques for Interpretable Machine Learningcited by 237
  34. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  35. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  36. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  37. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  38. 2015Doctor AI: Predicting Clinical Events via Recurrent Neural Networkscited by 205

50 most cited of 134 All 134 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 1.

11 papers.