# Outliers Dimensions that Disrupt Transformers Are Driven by Frequency

arXiv: https://arxiv.org/abs/2205.11380 · HTML: https://mlsanity.com/paper/2205.11380/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Understanding and Overcoming the Challenges of Efficient Transformer Quantization](https://mlsanity.com/paper/2109.12948/) (2021)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [When BERT Plays the Lottery, All Tickets Are Winning](https://mlsanity.com/paper/2005.00561/) (2020)
- [BERT Busters: Outlier Dimensions that Disrupt Transformers](https://mlsanity.com/paper/2105.06990/) (2021)
- [The MultiBERTs: BERT Reproductions for Robustness Analysis](https://mlsanity.com/paper/2106.16163/) (2021)
- [Positional Artefacts Propagate Through Masked Language Model Embeddings](https://mlsanity.com/paper/2011.04393/) (2020)
- [How is BERT surprised? Layerwise detection of linguistic anomalies](https://mlsanity.com/paper/2105.07452/) (2021)
- [Learning to Remove: Towards Isotropic Pre-trained BERT Embedding](https://mlsanity.com/paper/2104.05274/) (2021)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)

## Built on by (most cited first, 30 of 55)

- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling](https://mlsanity.com/paper/2304.09145/) (2023)
- [Intriguing Properties of Quantization at Scale](https://mlsanity.com/paper/2305.19268/) (2023)
- [Scalable Influence and Fact Tracing for Large Language Model Pretraining](https://mlsanity.com/paper/2410.17413/) (2024)
- [AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models](https://mlsanity.com/paper/2410.10912/) (2024)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps](https://mlsanity.com/paper/2302.00456/) (2023)
- [VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers](https://mlsanity.com/paper/2305.13417/) (2023)
- [Hyperbolic Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2410.04010/) (2024)
- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization](https://mlsanity.com/paper/2404.03605/) (2024)
- [A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training](https://mlsanity.com/paper/2601.22966/) (2026)
- [Unveiling Linguistic Regions in Large Language Models](https://mlsanity.com/paper/2402.14700/) (2024)
- [Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck](https://mlsanity.com/paper/2404.07647/) (2024)
- [Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs](https://mlsanity.com/paper/2405.14428/) (2024)
- [Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models](https://mlsanity.com/paper/2305.17446/) (2023)
- [Robust AI-Generated Text Detection by Restricted Embeddings](https://mlsanity.com/paper/2410.08113/) (2024)
- [Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity](https://mlsanity.com/paper/2306.00458/) (2023)
- [Merging Text Transformer Models from Different Initializations](https://mlsanity.com/paper/2403.00986/) (2024)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings](https://mlsanity.com/paper/2502.00919/) (2025)
- [Outlier Dimensions Encode Task-Specific Knowledge](https://mlsanity.com/paper/2310.17715/) (2023)
- [Understanding and Minimising Outlier Features in Neural Network Training](https://mlsanity.com/paper/2405.19279/) (2024)
- [Metis: Training LLMs with FP4 Quantization](https://mlsanity.com/paper/2509.00404/) (2025)
- [Outlier-weighed Layerwise Sampling for LLM Fine-tuning](https://mlsanity.com/paper/2405.18380/) (2024)

## Abstract (arXiv)

While Transformer-based language models are generally very robust to pruning, there is the recently discovered outlier phenomenon: disabling only 48 out of 110M parameters in BERT-base drops its performance by nearly 30% on MNLI. We replicate the original evidence for the outlier phenomenon and we link it to the geometry of the embedding space. We find that in both BERT and RoBERTa the magnitude of hidden state coefficients corresponding to outlier dimensions correlates with the frequency of encoded tokens in pre-training data, and it also contributes to the "vertical" self-attention pattern enabling the model to focus on the special tokens. This explains the drop in performance from disabling the outliers, and it suggests that to decrease anisotropicity in future models we need pre-training schemas that would better take into account the skewed token distributions.