mlsanityData to 14 Oct 2026

A Primer on the Inner Workings of Transformer-based Language Models

arXiv 2405.00208 ↗First posted 30 Apr 2024cs.CLcited by 85

Not on the map. Why?

Cited by 23 in Mechanistic interpretability, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive overview of the known internal mechanisms implemented by these models, uncovering connections across popular approaches and active research directions in this area.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 198 of its 410 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

198 of 410 references.

References · 198

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,384
38 more
  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2019Root Mean Square Layer Normalizationcited by 1,480
  9. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  10. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  13. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  14. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  15. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  16. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  17. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  18. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  19. 2022Mass-Editing Memory in a Transformercited by 1,001
  20. 2022Toy Models of Superpositioncited by 971
  21. 2022In-context Learning and Induction Headscited by 964
  22. 2019Attention is not Explanationcited by 929
  23. 2023Steering Language Models With Activation Engineeringcited by 922
  24. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  25. 2023Vision Transformers Need Registerscited by 871
  26. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  27. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  28. 2022Transformers learn in-context by gradient descentcited by 732
  29. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  30. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  31. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  32. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  33. 2016Language Modeling with Gated Convolutional Networkscited by 702
  34. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  35. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  36. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  37. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  38. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547

50 most cited of 198 All 198 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 28; 2025: 29; 2026: 28.

85 papers.

Sort:

By area:

  1. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  2. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  3. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  4. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  5. 2025Agentic Large Language Models, a surveycited by 108
  6. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 101
  7. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79
  8. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 72
  9. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
  10. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  11. 2024Confidence Regulation Neurons in Language Modelscited by 57
  12. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
38 more
  1. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  2. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Modelscited by 51
  3. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 45
  4. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  5. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 34
  6. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  7. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 28
  8. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  9. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  10. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 27
  11. 2024Can Editing LLMs Inject Harm?cited by 22
  12. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  13. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  14. 2024Residual Stream Analysis with Multi-Layer SAEscited by 19
  15. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  16. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  17. 2024Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questionscited by 15
  18. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelscited by 11
  19. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  20. 2025ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generationcited by 11
  21. 2025Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamicscited by 11
  22. 2025Universal Jailbreak Suffixes Are Strong Attention Hijackerscited by 10
  23. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMscited by 10
  24. 2025Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial Statescited by 9
  25. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 9
  26. 2026Differential syntactic and semantic encoding in LLMscited by 7
  27. 2024Order-Independence Without Fine Tuningcited by 6
  28. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  29. 2024Jet Expansions of Residual Computationcited by 4
  30. 2024Unlocking the Theory Behind Scaling 1-Bit Neural Networkscited by 4
  31. 2024Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation modelscited by 4
  32. 2025Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influencecited by 4
  33. 2026Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretabilitycited by 4
  34. 2026Automated Interpretability and Feature Discovery in Language Models with Agentscited by 4
  35. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  36. 2025Lightweight Latent Verifiers for Efficient Meta-Generation Strategiescited by 3
  37. 2025Localizing Persona Representations in LLMscited by 3
  38. 2025Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architecturescited by 3

50 most cited of 85 All 85 (JSON)