mlsanityData to 7 Oct 2026

AttentionViz: A Global View of Transformer Attention

arXiv 2305.03210 ↗First posted 4 May 2023cs.HC cs.CL cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Transformer models are revolutionizing machine learning, but their inner workings remain mysterious. In this work, we present a new visualization technique designed to help researchers understand the self-attention mechanism in transformers that allows these models to learn rich, contextual relationships between elements of a sequence. The main idea behind our method is to visualize a joint embedding of the query and key vectors used by transformer models to compute attention. Unlike previous attention visualization techniques, our approach enables the analysis of global patterns across multiple input sequences. We create an interactive visualization tool, AttentionViz (demo: http://attentionviz.com), based on these joint query-key embeddings, and use it to study attention mechanisms in both language and vision transformers. We demonstrate the utility of our approach in improving model understanding and offering new insights about query-key interactions through several application scenarios and expert feedback.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

51 of 70 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
38 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2022Toy Models of Superpositioncited by 928
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  7. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  8. 2021Intriguing Properties of Vision Transformerscited by 446
  9. 2022How Do Vision Transformers Work?cited by 305
  10. 2019Revealing the Dark Secrets of BERTcited by 300
  11. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  12. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  13. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  14. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  15. 2020TransPose: Keypoint Localization via Transformercited by 126
  16. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  17. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  18. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  19. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  20. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  21. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  22. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  23. 2019Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizationscited by 72
  24. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  25. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  26. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  27. 2016Embedding Projector: Interactive Visualization and Interpretation of Embeddingscited by 43
  28. 2019Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiplescited by 38
  29. 2019exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Modelscited by 33
  30. 2018A Comparison of Word Embeddings for the Biomedical Natural Language Processingcited by 29
  31. 2021Dodrio: Exploring Transformer Models with Interactive Visualizationcited by 28
  32. 2022Visualizing and Understanding Patch Interactions in Vision Transformercited by 26
  33. 2020Linguistic Profiling of a Neural Language Modelcited by 23
  34. 2019SANVis: Visual Analytics for Understanding Self-Attention Networkscited by 21
  35. 2021VisQA: X-raying Vision and Language Reasoning in Transformerscited by 19
  36. 2018EmbeddingVis: A Visual Analytics Approach to Comparative Network Embedding Inspectioncited by 18
  37. 2022Emblaze: Illuminating Machine Learning Representations through Interactive Comparison of Embedding Spacescited by 16
  38. 2023Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attentioncited by 15

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 20; 2025: 15; 2026: 8.

50 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2024Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Directioncited by 46
  3. 2023Foundation Models Meet Visualizations: Challenges and Opportunitiescited by 31
  4. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15
  5. 2024Attention Mechanism and Context Modeling System for Text Mining Machine Translationcited by 15
  6. 2023Explainability of Vision Transformers: A Comprehensive Review and New Perspectivescited by 12
  7. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12
  8. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attackscited by 11
  9. 2024Transformer Explainer: Learning LLM Transformers with Interactive Visual Explanation and Experimentationcited by 9
  10. 2025The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer trainingcited by 9
  11. 2024Understanding Large Language Model Behaviors through Interactive Counterfactual Generation and Analysiscited by 8
  12. 2024Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracitycited by 8
38 more
  1. 2024JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Modelscited by 7
  2. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  3. 2024ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scalecited by 5
  4. 2024Improving Interpretability and Robustness for the Detection of AI-Generated Imagescited by 5
  5. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  6. 2023Visual Analytics for Efficient Image Exploration and User-Guided Image Captioningcited by 4
  7. 2024An In-depth Investigation of Sparse Rate Reduction in Transformer-like Modelscited by 4
  8. 2025Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methodscited by 4
  9. 2023Attention Visualizer Package: Revealing Word Importance for Deeper Insight into Encoder-Only Transformer Modelscited by 3
  10. 2024Visualization for Trust in Machine Learning Revisited: The State of the Field in 2023cited by 3
  11. 2024Extending Token Computation for LLM Reasoningcited by 3
  12. 2024Brain-like Functional Organization within Large Language Modelscited by 3
  13. 2025Model Science: getting serious about verification, explanation and control of AI systemscited by 3
  14. 2026Decomposing Query-Key Feature Interactions Using Contrastive Covariancescited by 3
  15. 2023DeepDecipher: Accessing and Investigating Neuron Activation in Large Language Modelscited by 2
  16. 2024SingVisio: Visual Analytics of Diffusion Model for Singing Voice Conversioncited by 2
  17. 2024Audio Explanation Synthesis with Generative Foundation Modelscited by 2
  18. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 2
  19. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2
  20. 2024SpiralMLP: A Lightweight Vision MLP Architecturecited by 1
  21. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Modelscited by 1
  22. 2024JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasetscited by 1
  23. 2025User Perception of Attention Visualizations: Effects on Interpretability Across Evidence-Based Medical Documentscited by 1
  24. 2025NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Modelscited by 1
  25. 2025VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Modelscited by 1
  26. 2025ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Modelscited by 1
  27. 2026Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Modelscited by 1
  28. 2026KEditVis: A Visual Analytics System for Knowledge Editing of Large Language Modelscited by 1
  29. 2026LLM Explainability with Counterfactual Chains and Causal Graphscited by 1
  30. 2024Approximate Nullspace Augmented Finetuning for Robust Vision Transformerscited by 0
  31. 2024Automated Trustworthiness Oracle Generation for Machine Learning Text Classifierscited by 0
  32. 2025Visual Attention Exploration in Vision-Based Mamba Modelscited by 0
  33. 2025PLEX: Perturbation-free Local Explanations for LLM-Based Text Classificationcited by 0
  34. 2025Medical Interpretability and Knowledge Maps of Large Language Modelscited by 0
  35. 2025ExplainableGuard: Interpretable Adversarial Defense for Large Language Models Using Chain-of-Thought Reasoningcited by 0
  36. 2026CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generationcited by 0
  37. 2026VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphscited by 0
  38. 2026ASAP: Visual Analytics for Identifying and Analyzing Image Patterns in AI-generated Imagescited by 0