# VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

arXiv: https://arxiv.org/abs/2305.13417 · HTML: https://mlsanity.com/paper/2305.13417/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [BERT Busters: Outlier Dimensions that Disrupt Transformers](https://mlsanity.com/paper/2105.06990/) (2021)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Outliers Dimensions that Disrupt Transformers Are Driven by Frequency](https://mlsanity.com/paper/2205.11380/) (2022)
- [Attention Flows are Shapley Value Explanations](https://mlsanity.com/paper/2105.14652/) (2021)
- [Analyzing And Editing Inner Mechanisms Of Backdoored Language Models](https://mlsanity.com/paper/2302.12461/) (2023)

## Built on by (most cited first, 30 of 32)

- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Knowledge Circuits in Pretrained Transformers](https://mlsanity.com/paper/2405.17969/) (2024)
- [Finding Transformer Circuits with Edge Pruning](https://mlsanity.com/paper/2406.16778/) (2024)
- [Confidence Regulation Neurons in Language Models](https://mlsanity.com/paper/2406.16254/) (2024)
- [Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions](https://mlsanity.com/paper/2407.15018/) (2024)
- [Backward Lens: Projecting Language Model Gradients into the Vocabulary Space](https://mlsanity.com/paper/2402.12865/) (2024)
- [Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines](https://mlsanity.com/paper/2403.05846/) (2024)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Optimal ablation for interpretability](https://mlsanity.com/paper/2409.09951/) (2024)
- [Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models](https://mlsanity.com/paper/2502.10835/) (2025)
- [Revealing the Barriers of Language Agents in Planning](https://mlsanity.com/paper/2410.12409/) (2024)
- [LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models](https://mlsanity.com/paper/2404.07004/) (2024)
- [How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence](https://mlsanity.com/paper/2504.02904/) (2025)
- [Transformer Block Coupling and its Correlation with Generalization in LLMs](https://mlsanity.com/paper/2407.07810/) (2024)
- [Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning](https://mlsanity.com/paper/2502.04667/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [InversionView: A General-Purpose Method for Reading Information from Neural Activations](https://mlsanity.com/paper/2405.17653/) (2024)
- [Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability](https://mlsanity.com/paper/2507.02559/) (2025)
- [AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling](https://mlsanity.com/paper/2507.08567/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)
- [Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models](https://mlsanity.com/paper/2408.15091/) (2024)
- [Identifying and Mitigating the Influence of the Prior Distribution in Large Language Models](https://mlsanity.com/paper/2504.12585/) (2025)
- [Understanding Post-Training Structural Changes in Large Language Models](https://mlsanity.com/paper/2509.17866/) (2025)
- [Understanding Language Model Circuits through Knowledge Editing](https://mlsanity.com/paper/2406.17241/) (2024)
- [Reversed Attention: On The Gradient Descent Of Attention Layers In GPT](https://mlsanity.com/paper/2412.17019/) (2024)
- [Know-MRI: A Knowledge Mechanisms Revealer&Interpreter for Large Language Models](https://mlsanity.com/paper/2506.08427/) (2025)
- [NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models](https://mlsanity.com/paper/2509.03985/) (2025)
- [Context Copying Modulation: The Role of Entropy Neurons in Managing Parametric and Contextual Knowledge Conflicts](https://mlsanity.com/paper/2509.10663/) (2025)
- [RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models](https://mlsanity.com/paper/2604.19321/) (2026)

## Abstract (arXiv)

Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary, a transformation that makes them more human interpretable. In this paper, we investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mechanism. Based on our discoveries, we create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph, with nodes representing neurons or hidden states and edges representing the interactions between them. Our visualization simplifies huge amounts of data into easy-to-read plots that can reflect the models' internal processing, uncovering the contribution of each component to the models' final prediction. Our visualization also unveils new insights about the role of layer norms as semantic filters that influence the models' output, and about neurons that are always activated during forward passes and act as regularization vectors.