mlsanityData to 14 Oct 2026

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

arXiv 2305.13417 ↗First posted 22 May 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary, a transformation that makes them more human interpretable. In this paper, we investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mechanism. Based on our discoveries, we create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph, with nodes representing neurons or hidden states and edges representing the interactions between them. Our visualization simplifies huge amounts of data into easy-to-read plots that can reflect the models' internal processing, uncovering the contribution of each component to the models' final prediction. Our visualization also unveils new insights about the role of layer norms as semantic filters that influence the models' output, and about neurons that are always activated during forward passes and act as regularization vectors.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

21 of 42 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2016Layer Normalizationcited by 5,669
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  8. 2022Mass-Editing Memory in a Transformercited by 1,001
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2020Quantifying Attention Flow in Transformerscited by 775
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
9 more
  1. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  2. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
  3. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  4. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
  5. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  6. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  7. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  8. 2021Attention Flows are Shapley Value Explanationscited by 30
  9. 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 13; 2026: 3.

32 papers.

Sort:

By area:

  1. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  2. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  3. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  4. 2024Confidence Regulation Neurons in Language Modelscited by 57
  5. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  6. 2024Backward Lens: Projecting Language Model Gradients into the Vocabulary Spacecited by 32
  7. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 30
  8. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  9. 2024Optimal ablation for interpretabilitycited by 25
  10. 2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
  11. 2024Revealing the Barriers of Language Agents in Planningcited by 20
  12. 2024LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Modelscited by 16
20 more
  1. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
  2. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 10
  3. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  4. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  5. 2025HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMscited by 9
  6. 2024InversionView: A General-Purpose Method for Reading Information from Neural Activationscited by 7
  7. 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretabilitycited by 7
  8. 2025AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modelingcited by 7
  9. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  10. 2024Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Modelscited by 4
  11. 2025Identifying and Mitigating the Influence of the Prior Distribution in Large Language Modelscited by 4
  12. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  13. 2024Understanding Language Model Circuits through Knowledge Editingcited by 2
  14. 2024Reversed Attention: On The Gradient Descent Of Attention Layers In GPTcited by 2
  15. 2025Know-MRI: A Knowledge Mechanisms Revealer&Interpreter for Large Language Modelscited by 1
  16. 2025NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Modelscited by 1
  17. 2025Context Copying Modulation: The Role of Entropy Neurons in Managing Parametric and Contextual Knowledge Conflictscited by 1
  18. 2026RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Modelscited by 0
  19. 2026Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Modelscited by 0
  20. 2026The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priorscited by 0