# SANVis: Visual Analytics for Understanding Self-Attention Networks

arXiv: https://arxiv.org/abs/1909.09595 · HTML: https://mlsanity.com/paper/1909.09595/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2019.

## Built on by (most cited first, 21 of 21)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Revisiting Over-smoothing in BERT from the Perspective of Graph](https://mlsanity.com/paper/2202.08625/) (2022)
- [AttentionViz: A Global View of Transformer Attention](https://mlsanity.com/paper/2305.03210/) (2023)
- [Identifying Semantic Induction Heads to Understand In-Context Learning](https://mlsanity.com/paper/2402.13055/) (2024)
- [SparseBERT: Rethinking the Importance Analysis in Self-attention](https://mlsanity.com/paper/2102.12871/) (2021)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [How Does Attention Work in Vision Transformers? A Visual Analytics Attempt](https://mlsanity.com/paper/2303.13731/) (2023)
- [InternalInspector I2: Robust Confidence Estimation in LLMs through Internal States](https://mlsanity.com/paper/2406.12053/) (2024)
- [Visual Fourier Prompt Tuning](https://mlsanity.com/paper/2411.01327/) (2024)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations](https://mlsanity.com/paper/2411.12701/) (2024)
- [Visual Comparison of Language Model Adaptation](https://mlsanity.com/paper/2208.08176/) (2022)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [RECAST: Enabling User Recourse and Interpretability of Toxicity Detection Models with Interactive Visualization](https://mlsanity.com/paper/2102.04427/) (2021)
- [A Visual Analytics System for Improving Attention-based Traffic Forecasting Models](https://mlsanity.com/paper/2208.04350/) (2022)
- [Finspector: A Human-Centered Visual Inspection Tool for Exploring and Comparing Biases among Foundation Models](https://mlsanity.com/paper/2305.16937/) (2023)
- [ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale](https://mlsanity.com/paper/2404.02990/) (2024)
- [Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning](https://mlsanity.com/paper/2311.01016/) (2023)
- [k* Distribution: Evaluating the Latent Space of Deep Neural Networks using Local Neighborhood Analysis](https://mlsanity.com/paper/2312.04024/) (2023)
- [JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasets](https://mlsanity.com/paper/2407.03045/) (2024)
- [Visual Attention Exploration in Vision-Based Mamba Models](https://mlsanity.com/paper/2502.20764/) (2025)

## Abstract (arXiv)

Attention networks, a deep neural network architecture inspired by humans' attention mechanism, have seen significant success in image captioning, machine translation, and many other applications. Recently, they have been further evolved into an advanced approach called multi-head self-attention networks, which can encode a set of input vectors, e.g., word vectors in a sentence, into another set of vectors. Such encoding aims at simultaneously capturing diverse syntactic and semantic features within a set, each of which corresponds to a particular attention head, forming altogether multi-head attention. Meanwhile, the increased model complexity prevents users from easily understanding and manipulating the inner workings of models. To tackle the challenges, we present a visual analytics system called SANVis, which helps users understand the behaviors and the characteristics of multi-head self-attention networks. Using a state-of-the-art self-attention model called Transformer, we demonstrate usage scenarios of SANVis in machine translation tasks. Our system is available at http://short.sanvis.org