# Influence Patterns for Explaining Information Flow in BERT

arXiv: https://arxiv.org/abs/2011.00740 · HTML: https://mlsanity.com/paper/2011.00740/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 14 of 14)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Measuring the Mixing of Contextual Information in the Transformer](https://mlsanity.com/paper/2203.04212/) (2022)
- [Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations](https://mlsanity.com/paper/2403.18167/) (2024)
- [Transformer Explainer: Learning LLM Transformers with Interactive Visual Explanation and Experimentation](https://mlsanity.com/paper/2408.04619/) (2024)
- [Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models](https://mlsanity.com/paper/2511.06793/) (2025)
- [Discovering Influential Neuron Path in Vision Transformers](https://mlsanity.com/paper/2503.09046/) (2025)
- [SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models](https://mlsanity.com/paper/2510.05173/) (2025)
- [Generalizing Backpropagation for Gradient-Based Interpretability](https://mlsanity.com/paper/2307.03056/) (2023)
- [AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning](https://mlsanity.com/paper/2210.05883/) (2022)
- [Validating Mechanistic Interpretations: An Axiomatic Approach](https://mlsanity.com/paper/2407.13594/) (2024)
- [Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs](https://mlsanity.com/paper/2604.13258/) (2026)
- [Order-sensitive Shapley Values for Evaluating Conceptual Soundness of NLP Models](https://mlsanity.com/paper/2206.00192/) (2022)
- [Breaking BERT: Evaluating and Optimizing Sparsified Attention](https://mlsanity.com/paper/2210.03841/) (2022)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)

## Abstract (arXiv)

While attention is all you need may be proving true, we do not know why: attention-based transformer models such as BERT are superior but how information flows from input tokens to output predictions are unclear. We introduce influence patterns, abstractions of sets of paths through a transformer model. Patterns quantify and localize the flow of information to paths passing through a sequence of model nodes. Experimentally, we find that significant portion of information flow in BERT goes through skip connections instead of attention heads. We further show that consistency of patterns across instances is an indicator of BERT's performance. Finally, We demonstrate that patterns account for far more model performance than previous attention-based and layer-based methods.