mlsanityData to 7 Oct 2026

All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

arXiv 2109.04404 ↗First posted 9 Sep 2021cs.CL cs.LGcited by 150

Not on the map. Why?

Abstract (arXiv)

Similarity measures are a vital tool for understanding how language models represent and process language. Standard representational similarity measures such as cosine similarity and Euclidean distance have been successfully used in static word embedding models to understand how words cluster in semantic space. Recently, these measures have been applied to embeddings from contextualized models such as BERT and GPT-2. In this work, we call into question the informativity of such measures for contextualized language models. We find that a small number of rogue dimensions, often just 1-3, dominate these measures. Moreover, we find a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model. We show that simple postprocessing techniques such as standardization are able to correct for rogue dimensions and reveal underlying representational quality. We argue that accounting for rogue dimensions is essential for any similarity-based analysis of contextual language models.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

28 of 47 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  10. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  11. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  12. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
16 more
  1. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  2. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  3. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  4. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  5. 2018On the importance of single directions for generalizationcited by 99
  6. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  7. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  8. 2016SimVerb-3500: A Large-Scale Evaluation Set of Verb Similaritycited by 44
  9. 2021Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Sensescited by 39
  10. 2019Evaluating Word Embedding Models: Methods and Experimental Resultscited by 37
  11. 2019Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brainscited by 37
  12. 2019Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shiftcited by 37
  13. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 34
  14. 2019Correlating neural and symbolic representations of languagecited by 32
  15. 2019Correlation Coefficients and Semantic Textual Similaritycited by 20
  16. 2020Discourse structure interacts with reference but not syntax in neural language modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 17; 2023: 28; 2024: 30; 2025: 16; 2026: 57.

150 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  4. 2024Massive Activations in Large Language Modelscited by 277
  5. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  6. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  7. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  8. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  9. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  10. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  11. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  12. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
38 more
  1. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  2. 2023Quantifying Context Mixing in Transformerscited by 43
  3. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  4. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  5. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  6. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  7. 2022Neural reality of argument structure constructionscited by 25
  8. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  9. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  10. 2023Evaluating Biased Attitude Associations of Language Models in an Intersectional Contextcited by 20
  11. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 19
  12. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  13. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  14. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 19
  15. 2022Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddingscited by 18
  16. 2024ReSi: A Comprehensive Benchmark for Representational Similarity Measurescited by 18
  17. 2024Robust AI-Generated Text Detection by Restricted Embeddingscited by 18
  18. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  19. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  20. 2023Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lenscited by 15
  21. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  22. 2022Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representationscited by 14
  23. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  24. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  25. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  26. 2023How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structurecited by 13
  27. 2022Discovering Differences in the Representation of People using Contextualized Semantic Axescited by 12
  28. 2023Stable Anisotropic Regularizationcited by 12
  29. 2023Investigating semantic subspaces of Transformer sentence embeddings through linear structural probingcited by 12
  30. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 12
  31. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  32. 2024Can Large Language Models Understand Molecules?cited by 10
  33. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  34. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 9
  35. 2022Combining Static and Contextualised Multilingual Embeddingscited by 8
  36. 2022Isotropic Representation Can Improve Dense Retrievalcited by 8
  37. 2023Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Modelscited by 8
  38. 2023Representation Of Lexical Stylistic Features In Language Models' Embedding Spacecited by 8

50 most cited of 150 All 150 (JSON)