mlsanityData to 7 Oct 2026

Towards Uncovering How Large Language Model Works: An Explainability Perspective

arXiv 2402.10688 ↗First posted 16 Feb 2024cs.CLcited by 22

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have led to breakthroughs in language tasks, yet the internal mechanisms that enable their remarkable generalization and reasoning abilities remain opaque. This lack of transparency presents challenges such as hallucinations, toxicity, and misalignment with human values, hindering the safe and beneficial deployment of LLMs. This paper aims to uncover the mechanisms underlying LLM functionality through the lens of explainability. First, we review how knowledge is architecturally composed within LLMs and encoded in their internal parameters via mechanistic interpretability techniques. Then, we summarize how knowledge is embedded in LLM representations by leveraging probing techniques and representation engineering. Additionally, we investigate the training dynamics through a mechanistic perspective to explain phenomena such as grokking and memorization. Lastly, we explore how the insights gained from these explanations can enhance LLM performance through model editing, improve efficiency through pruning, and better align with human values.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 47 of its 71 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

47 of 71 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
35 more
  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2022In-context Learning and Induction Headscited by 930
  3. 2023Steering Language Models With Activation Engineeringcited by 824
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  11. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  12. 2023Explainability for Large Language Models: A Surveycited by 331
  13. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  14. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  15. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  16. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  17. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  18. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  19. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  20. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  21. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  22. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  23. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  24. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  25. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  26. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  27. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  28. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  29. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  30. 2023Learning Transformer Programscited by 40
  31. 2023Grokking in Linear Estimators -- A Solvable Model that Groks without Understandingcited by 25
  32. 2023To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasetscited by 24
  33. 2024Critical Data Size of Language Models from a Grokking Perspectivecited by 22
  34. 2023What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causescited by 19
  35. 2023Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Headscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 6.

22 papers.

Sort:

By area:

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  2. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  4. 2025Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamicscited by 16
  5. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  6. 2025Causality Matters: How Temporal Information Emerges in Video Language Modelscited by 9
  7. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  8. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  9. 2024Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neurosciencecited by 5
  10. 2026Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantificationcited by 5
  11. 2024Interpretable Machine Learning for Weather and Climate Prediction: A Surveycited by 4
  12. 2024Generating a Low-code Complete Workflow via Task Decomposition and RAGcited by 3
10 more
  1. 2026Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMscited by 3
  2. 2025Interpreting the Latent Structure of Operator Precedence in Language Modelscited by 2
  3. 2025Interpreting and Mitigating Unwanted Uncertainty in LLMscited by 2
  4. 2026Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomycited by 2
  5. 2026Mechanistic Interpretability Tool for AI Weather Modelscited by 2
  6. 2024Parameter Choice and Neuro-Symbolic Approaches for Deep Domain-Invariant Learningcited by 1
  7. 2025Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokkingcited by 1
  8. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agentscited by 1
  9. 2025From Parameters to Performance: A Data-Driven Study on LLM Structure and Developmentcited by 0
  10. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  2. 2024Rethinking Interpretability in the Era of Large Language Models
  3. 2025Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
  4. 2026Understanding Large Language Models
  5. 2024LLMs' Understanding of Natural Language Revealed
  6. 2026Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions
  7. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
  8. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models