# Attention Visualizer Package: Revealing Word Importance for Deeper Insight into Encoder-Only Transformer Models

arXiv: https://arxiv.org/abs/2308.14850 · HTML: https://mlsanity.com/paper/2308.14850/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [AttentionViz: A Global View of Transformer Attention](https://mlsanity.com/paper/2305.03210/) (2023)
- [Enhancing Explainability of Neural Networks through Architecture Constraints](https://mlsanity.com/paper/1901.03838/) (2019)

## Built on by (most cited first, 3 of 3)

- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences](https://mlsanity.com/paper/2502.20478/) (2025)

## Abstract (arXiv)

This report introduces the Attention Visualizer package, which is crafted to visually illustrate the significance of individual words in encoder-only transformer-based models. In contrast to other methods that center on tokens and self-attention scores, our approach will examine the words and their impact on the final embedding representation. Libraries like this play a crucial role in enhancing the interpretability and explainability of neural networks. They offer the opportunity to illuminate their internal mechanisms, providing a better understanding of how they operate and can be enhanced. You can access the code and review examples on the following GitHub repository: https://github.com/AlaFalaki/AttentionVisualizer.