Explainability in Practice: A Survey of Explainable NLP Across Various Domains
Not on the map. Why?
Abstract (arXiv)
Natural Language Processing (NLP) is now embedded in critical sectors including healthcare, finance, and customer relationship management, where models such as GPT-4o, Gemini, and BERT increasingly inform decisions. The black-box nature of these models has created an urgent need for transparency. This review examines explainable NLP (XNLP) as it is actually deployed, working through seven application domains: medicine, finance, systematic reviews, customer relationship management, chatbots, social and behavioral science, and human resources. For each domain, we ask what kind of explanation the setting needs, which methods are used there, and how they are evaluated. A structured cross-domain synthesis then contrasts how those requirements diverge. We compare the main explanation method families on scope, evidence of faithfulness, and computational cost. We also propose a two-tier evaluation protocol that separates a shared technical core of metrics from the domain-specific validation layer through which those metrics have to be read. The review also addresses areas that remain underrepresented in the XNLP literature, including real-world applicability, the gap between fidelity and faithfulness, and the role of human judgment in assessing explanations. It closes with research directions, among them personalized explanations, human-in-the-loop evaluation, and mechanistic interpretability for large language models.
Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 134 of its 285 references that are in the map. First posted on arXiv 2 Feb 2025.
Builds on
134 of 285 references.
References · 134
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
- 2017Axiomatic Attribution for Deep Networkscited by 3,350
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
38 more
- 2023A Survey of Large Language Modelscited by 2,537
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
- 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
- 2018Sanity Checks for Saliency Mapscited by 1,149
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
- 2019Attention is not Explanationcited by 916
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
- 2019Attention is not not Explanationcited by 629
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
- 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
- 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
- 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
- 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
- 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
- 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
- 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
- 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 344
- 2023Explainability for Large Language Models: A Surveycited by 331
- 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
- 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
- 2023Faithful Chain-of-Thought Reasoningcited by 285
- 2016Rationalizing Neural Predictionscited by 281
- 2018Metrics for Explainable AI: Challenges and Prospectscited by 257
- 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
- 2018Techniques for Interpretable Machine Learningcited by 237
- 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
- 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
- 2016Model-Agnostic Interpretability of Machine Learningcited by 219
- 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
- 2015Doctor AI: Predicting Clinical Events via Recurrent Neural Networkscited by 205
50 most cited of 134 All 134 (JSON)
Built on by
Citing papers per year: 2025: 10; 2026: 1.
11 papers.
- 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologiescited by 17
- 2025ADVICE: Answer-Dependent Verbalized Confidence Estimationcited by 8
- 2025EvalMORAAL: Interpretable Chain-of-Thought and LLM-as-Judge Evaluation for Moral Alignment in Large Language Modelscited by 6
- 2025Do Large Language Models Understand Morality Across Cultures?cited by 4
- 2025Exploring Cultural Variations in Moral Judgments with Large Language Modelscited by 3
- 2025Towards Transparent AI: A Survey on Explainable Language Modelscited by 3
- 2025Concept Layers: Enhancing Interpretability and Intervenability via LLM Conceptualizationcited by 2
- 2025Explainability-Based Token Replacement on LLM-Generated Textcited by 2
- 2025Conceptual Contrastive Edits in Textual and Vision-Language Retrievalcited by 0
- 2025A Computational Framework to Identify Self-Aspects in Textcited by 0
- 2026Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillationcited by 0
- 2026Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation11 Mar 2026
- 2025ADVICE: Answer-Dependent Verbalized Confidence Estimation13 Oct 2025
- 2025EvalMORAAL: Interpretable Chain-of-Thought and LLM-as-Judge Evaluation for Moral Alignment in Large Language Models7 Oct 2025
- 2025Towards Transparent AI: A Survey on Explainable Language Models25 Sep 2025
- 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies14 Aug 2025
- 2025Do Large Language Models Understand Morality Across Cultures?28 Jul 2025
- 2025A Computational Framework to Identify Self-Aspects in Text17 Jul 2025
- 2025Exploring Cultural Variations in Moral Judgments with Large Language Models14 Jun 2025
- 2025Explainability-Based Token Replacement on LLM-Generated Text4 Jun 2025
- 2025Conceptual Contrastive Edits in Textual and Vision-Language Retrieval1 Mar 2025
- 2025Concept Layers: Enhancing Interpretability and Intervenability via LLM Conceptualization19 Feb 2025