mlsanityData to 14 Oct 2026

Explainability for Large Language Models: A Survey

arXiv 2309.01029 ↗First posted 2 Sep 2023cs.CL cs.AI cs.LGcited by 333

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their internal mechanisms are still unclear and this lack of transparency poses unwanted risks for downstream applications. Therefore, understanding and explaining these models is crucial for elucidating their behaviors, limitations, and social impacts. In this paper, we introduce a taxonomy of explainability techniques and provide a structured overview of methods for explaining Transformer-based language models. We categorize techniques based on the training paradigms of LLMs: traditional fine-tuning-based paradigm and prompting-based paradigm. For each paradigm, we summarize the goals and dominant approaches for generating local explanations of individual predictions and global explanations of overall model knowledge. We also discuss metrics for evaluating generated explanations, and discuss how explanations can be leveraged to debug models and improve performance. Lastly, we examine key challenges and emerging opportunities for explanation techniques in the era of LLMs in comparison to conventional machine learning models.

Key facts. Cited by 333 papers in the map (data through 14 Oct 2026). Builds on 159 of its 217 references that are in the map. First posted on arXiv 2 Sep 2023.

Builds on

159 of 217 references.

References · 159

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  10. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
  13. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  14. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  15. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  16. 2021Ethical and social risks of harm from Language Modelscited by 966
  17. 2019Attention is not Explanationcited by 929
  18. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  19. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  20. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  21. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  22. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  23. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  24. 2019Designing and Interpreting Probes with Control Taskscited by 640
  25. 2019Attention is not not Explanationcited by 636
  26. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  27. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  28. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  29. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  30. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  31. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  32. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  33. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  34. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  35. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  36. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  37. 2023Larger language models do in-context learning differentlycited by 394
  38. 2019Is Attention Interpretable?cited by 389

50 most cited of 159 All 159 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 135; 2025: 133; 2026: 46.

333 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  4. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  6. 2024Tool Learning with Large Language Models: A Surveycited by 223
  7. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  8. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  9. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  10. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  11. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  12. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
38 more
  1. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 98
  2. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  3. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  4. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 85
  5. 2024Prover-Verifier Games improve legibility of LLM outputscited by 85
  6. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 83
  7. 2024NetLLM: Adapting Large Language Models for Networkingcited by 78
  8. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 73
  9. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  10. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 65
  11. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  12. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 63
  13. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  14. 2024Determinants of LLM-assisted Decision-Makingcited by 56
  15. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 56
  16. 2024Language Repository for Long Video Understandingcited by 55
  17. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  18. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 53
  19. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  20. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  21. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  22. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  23. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 48
  24. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  25. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  26. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  27. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  28. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  29. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 34
  30. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  31. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  32. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 30
  33. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  34. 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulationscited by 28
  35. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 27
  36. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  37. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  38. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 26

50 most cited of 333 All 333 (JSON)