mlsanityData to 7 Oct 2026

XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models

arXiv 2407.15248 ↗First posted 21 Jul 2024cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

In this survey, we address the key challenges in Large Language Models (LLM) research, focusing on the importance of interpretability. Driven by increasing interest from AI and business sectors, we highlight the need for transparency in LLMs. We examine the dual paths in current LLM research and eXplainable Artificial Intelligence (XAI): enhancing performance through XAI and the emerging focus on model interpretability. Our paper advocates for a balanced approach that values interpretability equally with functional advancements. Recognizing the rapid development in LLM research, our survey includes both peer-reviewed and preprint (arXiv) papers, offering a comprehensive overview of XAI's role in LLM research. We conclude by urging the research community to advance both LLM and XAI fields together.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 45 of its 70 references that are in the map. First posted on arXiv 21 Jul 2024.

Builds on

45 of 70 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  6. 2021Ethical and social risks of harm from Language Modelscited by 957
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  9. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  10. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  12. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
33 more
  1. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  2. 2022Visual Classification via Description from Large Language Modelscited by 331
  3. 2023Explainability for Large Language Models: A Surveycited by 331
  4. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  5. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  6. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  7. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  8. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  9. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  10. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  11. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  12. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  13. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  14. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  15. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  16. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  17. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  18. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  19. 2023Eight Things to Know about Large Language Modelscited by 73
  20. 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluationcited by 69
  21. 2023Explaining black box text modules in natural language with language modelscited by 61
  22. 2023The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challengescited by 59
  23. 2023Post Hoc Explanations of Language Models Can Improve Language Modelscited by 50
  24. 2023Explainable Automated Debugging via Large Language Model-driven Scientific Debuggingcited by 40
  25. 2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Mapscited by 35
  26. 2023ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphscited by 27
  27. 2024Generative AI in EU Law: Liability, Privacy, Intellectual Property, and Cybersecuritycited by 21
  28. 2023Improved Logical Reasoning of Language Models via Differentiable Symbolic Programmingcited by 18
  29. 2021Interpreting Language Models Through Knowledge Graph Extractioncited by 14
  30. 2023A Comprehensive Review on Financial Explainable AIcited by 10
  31. 2023The Importance of Human-Labeled Data in the Era of LLMscited by 8
  32. 2023Can ChatGPT's Responses Boost Traditional Natural Language Processing?cited by 6
  33. 2023TbExplain: A Text-based Explanation Method for Scene Classification Models with the Statistical Prediction Correctioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 5.

23 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  5. 2025Layered Chain-of-Thought Prompting for Multi-Agent LLM Systems: A Comprehensive Approach to Explainable Large Language Modelscited by 10
  6. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  7. 2025Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Modelscited by 4
  8. 2024Negation Blindness in Large Language Models: Unveiling the NO Syndrome in Image Generationcited by 3
  9. 2025Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Modelscited by 3
  10. 2026Explainable AI in Speaker Recognition -- Making Latent Representations Understandablecited by 3
  11. 2024Towards Secure Intelligent O-RAN Architecture: Vulnerabilities, Threats and Promising Technical Solutions using LLMscited by 2
  12. 2025Envisioning an AI-Enhanced Mental Health Ecosystemcited by 2
11 more
  1. 2025Explain First, Trust Later: LLM-Augmented Explanations for Graph-Based Crypto Anomaly Detectioncited by 2
  2. 2025Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedbackcited by 2
  3. 2025Bridging Minds and Machines: Toward an Integration of AI and Cognitive Sciencecited by 2
  4. 2025Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Modelscited by 2
  5. 2026Reasoning Fails Where Step Flow Breakscited by 2
  6. 2025Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Modelscited by 1
  7. 2025Triadic Fusion of Cognitive, Functional, and Causal Dimensions for Explainable LLMs: The TAXAL Frameworkcited by 1
  8. 2025Mitigating Clinician Information Overload: Generative AI for Integrated EHR and RPM Data Analysiscited by 0
  9. 2026Analysis and Explainability of LLMs Via Evolutionary Methodscited by 0
  10. 2026Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Modelscited by 0
  11. 2026Temporally Interpretable Differentiable Decision Treescited by 0