mlsanityData to 7 Oct 2026

From Understanding to Utilization: A Survey on Explainability for Large Language Models

arXiv 2401.12874 ↗First posted 23 Jan 2024cs.CL cs.AIcited by 47

Not on the map. Why?

Abstract (arXiv)

Explainability for Large Language Models (LLMs) is a critical yet challenging aspect of natural language processing. As LLMs are increasingly integral to diverse applications, their "black-box" nature sparks significant concerns regarding transparency and ethical use. This survey underscores the imperative for increased explainability in LLMs, delving into both the research on explainability and the various methodologies and tasks that utilize an understanding of these models. Our focus is primarily on pre-trained Transformer-based LLMs, such as LLaMA family, which pose distinctive interpretability challenges due to their scale and complexity. In terms of existing methods, we classify them into local and global analyses, based on their explanatory objectives. When considering the utilization of explainability, we explore several compelling methods that concentrate on model editing, control generation, and model enhancement. Additionally, we examine representative evaluation metrics and datasets, elucidating their advantages and limitations. Our goal is to reconcile theoretical and empirical understanding with practical implementation, proposing exciting avenues for explanatory techniques and their applications in the LLMs era.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

52 of 72 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016Layer Normalizationcited by 5,579
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2019Language Models as Knowledge Bases?cited by 1,922
38 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2022Mass-Editing Memory in a Transformercited by 976
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2020Quantifying Attention Flow in Transformerscited by 759
  9. 2022A Survey on In-context Learningcited by 717
  10. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  13. 2022Memory-Based Model Editing at Scalecited by 443
  14. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  15. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  16. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  17. 2023Explainability for Large Language Models: A Surveycited by 331
  18. 2023In-Context Learning Creates Task Vectorscited by 302
  19. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  20. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  21. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  22. 2023PMET: Precise Model Editing in a Transformercited by 170
  23. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  24. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  25. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  26. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  27. 2022Analyzing Transformers in Embedding Spacecited by 131
  28. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  29. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  30. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  31. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  32. 2020Generating Hierarchical Explanations on Text Classification via Feature Interaction Detectioncited by 57
  33. 2016Investigating the influence of noise and distractors on the interpretation of neural networkscited by 54
  34. 2023Sequential Integrated Gradients: a simple but effective method for explaining language modelscited by 52
  35. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  36. 2022Grad-SAM: Explaining Transformers via Gradient Self-Attention Mapscited by 42
  37. 2023Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Mapscited by 33
  38. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 24; 2026: 9.

47 papers.

Sort:

By area:

  1. 2024Attention Heads of Large Language Models: A Surveycited by 65
  2. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  3. 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulationscited by 28
  4. 2024Finding Visual Task Vectorscited by 24
  5. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  6. 2026Large Language Model Reasoning Failurescited by 20
  7. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologiescited by 17
  8. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 16
  9. 2025An overview of model uncertainty and variability in LLM-based sentiment analysis. Challenges, mitigation strategies and the role of explainabilitycited by 15
  10. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  11. 2025Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American Englishcited by 8
  12. 2025Grounding LLM Reasoning with Knowledge Graphscited by 7
35 more
  1. 2025Large Language Models as Computable Approximations to Solomonoff Inductioncited by 6
  2. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  3. 2024Lightweight Safety Classification Using Pruned Language Modelscited by 5
  4. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  5. 2024Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Modelscited by 4
  6. 2024The Minimum Information about CLinical Artificial Intelligence Checklist for Generative Modeling Research (MI-CLAIM-GEN)cited by 3
  7. 2024Automated Multi-Label Annotation for Mental Health Illnesses Using Large Language Modelscited by 3
  8. 2025Transparent AI: The Case for Interpretability and Explainabilitycited by 3
  9. 2025Towards Transparent AI: A Survey on Explainable Language Modelscited by 3
  10. 2024Model Lakescited by 2
  11. 2024Through the Thicket: A Study of Number-Oriented LLMs derived from Random Forest Modelscited by 2
  12. 2025Can Large Language Models Simulate Human Responses? A Case Study of Stated Preference Experiments in the Context of Heating-related Choicescited by 2
  13. 2025Unleashing Diverse Thinking Modes in LLMs through Multi-Agent Collaborationcited by 2
  14. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  15. 2025Multi-agent Self-triage System with Medical Flowchartscited by 2
  16. 2026Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detectioncited by 2
  17. 2026Reasoning Fails Where Step Flow Breakscited by 2
  18. 2026Don't Lose Focus: Activation Steering via Key-Orthogonal Projectionscited by 2
  19. 2024A Theoretical Survey on Foundation Modelscited by 1
  20. 2024Tuning Language Models by Mixture-of-Depths Ensemblecited by 1
  21. 2025From Text to Graph: Leveraging Graph Neural Networks for Enhanced Explainability in NLPcited by 1
  22. 2025Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Usecited by 1
  23. 2025Composable Building Blocks for Controllable and Transparent Interactive AI Systemscited by 1
  24. 2025Triadic Fusion of Cognitive, Functional, and Causal Dimensions for Explainable LLMs: The TAXAL Frameworkcited by 1
  25. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  26. 2025CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updatescited by 1
  27. 2026METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Modelscited by 1
  28. 2026Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarkscited by 1
  29. 2026Probabilistic Attribution For Large Language Modelscited by 1
  30. 2024TinyEmo: Scaling down Emotional Reasoning via Metric Projectioncited by 0
  31. 2025PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relationscited by 0
  32. 2025xInv: Explainable Optimization of Inverse Problemscited by 0
  33. 2025CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentationcited by 0
  34. 2025MATCH: Engineering Transparent and Controllable Conversational XAI Systems through Composable Building Blockscited by 0
  35. 2026Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representationscited by 0