# exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

arXiv: https://arxiv.org/abs/1910.05276 · HTML: https://mlsanity.com/paper/1910.05276/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 30 of 33)

- [Benchmarking and Survey of Explanation Methods for Black Box Models](https://mlsanity.com/paper/2102.13076/) (2021)
- [Eliciting Latent Knowledge from Quirky Language Models](https://mlsanity.com/paper/2312.01037/) (2023)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)
- [VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers](https://mlsanity.com/paper/2203.17247/) (2022)
- [What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code](https://mlsanity.com/paper/2202.06840/) (2022)
- [AttentionViz: A Global View of Transformer Attention](https://mlsanity.com/paper/2305.03210/) (2023)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior](https://mlsanity.com/paper/2107.09234/) (2021)
- [LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models](https://mlsanity.com/paper/2204.12130/) (2022)
- [Visual Comparison of Language Model Adaptation](https://mlsanity.com/paper/2208.08176/) (2022)
- [GenNI: Human-AI Collaboration for Data-Backed Text Generation](https://mlsanity.com/paper/2110.10185/) (2021)
- [Interpreting BERT-based Text Similarity via Activation and Saliency Maps](https://mlsanity.com/paper/2208.06612/) (2022)
- [Neuron-based Personality Trait Induction in Large Language Models](https://mlsanity.com/paper/2410.12327/) (2024)
- [RECAST: Enabling User Recourse and Interpretability of Toxicity Detection Models with Interactive Visualization](https://mlsanity.com/paper/2102.04427/) (2021)
- [Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference](https://mlsanity.com/paper/2404.03085/) (2024)
- [Explainable AI for Mental Disorder Detection via Social Media: A survey and outlook](https://mlsanity.com/paper/2406.05984/) (2024)
- [It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers](https://mlsanity.com/paper/2204.11424/) (2022)
- [DeepLens: Interactive Out-of-distribution Data Detection in NLP Models](https://mlsanity.com/paper/2303.01577/) (2023)
- [Procedural Adherence and Interpretability Through Neuro-Symbolic Generative Agents](https://mlsanity.com/paper/2402.16905/) (2024)
- [Lightweight Safety Classification Using Pruned Language Models](https://mlsanity.com/paper/2412.13435/) (2024)
- [Reasoning-Grounded Natural Language Explanations for Language Models](https://mlsanity.com/paper/2503.11248/) (2025)
- [Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers](https://mlsanity.com/paper/2510.21986/) (2025)
- [Astroconformer: Inferring Surface Gravity of Stars from Stellar Light Curves with Transformer](https://mlsanity.com/paper/2207.02787/) (2022)
- [Visual Analytics for Generative Transformer Models](https://mlsanity.com/paper/2311.12418/) (2023)
- [Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task](https://mlsanity.com/paper/2209.03557/) (2022)
- [Opening the Black Box: Analyzing Attention Weights and Hidden States in Pre-trained Language Models for Non-language Tasks](https://mlsanity.com/paper/2306.12198/) (2023)
- [Consistency Matters: Explore LLMs Consistency From a Black-Box Perspective](https://mlsanity.com/paper/2402.17411/) (2024)
- [A Theoretical Survey on Foundation Models](https://mlsanity.com/paper/2410.11444/) (2024)
- [Representing ELMo embeddings as two-dimensional text online](https://mlsanity.com/paper/2103.16414/) (2021)

## Abstract (arXiv)

Large language models can produce powerful contextual representations that lead to improvements across many NLP tasks. Since these models are typically guided by a sequence of learned self attention mechanisms and may comprise undesired inductive biases, it is paramount to be able to explore what the attention has learned. While static analyses of these models lead to targeted insights, interactive tools are more dynamic and can help humans better gain an intuition for the model-internal reasoning process. We present exBERT, an interactive tool named after the popular BERT language model, that provides insights into the meaning of the contextual representations by matching a human-specified input to similar contexts in a large annotated dataset. By aggregating the annotations of the matching similar contexts, exBERT helps intuitively explain what each attention-head has learned.