mlsanityData to 7 Oct 2026

Anisotropy Is Inherent to Self-Attention in Transformers

arXiv 2401.12143 ↗First posted 22 Jan 2024cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations suggest that anisotropy is actually inherent to Transformers-based models.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

33 of 46 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
21 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  7. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  8. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  9. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  10. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  12. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  13. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  14. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  15. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  16. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  17. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  18. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  19. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 19
  20. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  21. 2021Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representationscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 35.

62 papers.

Sort:

By area:

  1. 2025On the Emergence of Position Bias in Transformerscited by 57
  2. 2024Transformer Layers as Painterscited by 50
  3. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  4. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  5. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  6. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  7. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
  8. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  9. 2025Once Upon a Time: Interactive Learning for Storytelling with Small Language Modelscited by 11
  10. 2024Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothingcited by 10
  11. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  12. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 7
38 more
  1. 2024MEXMA: Token-level objectives improve sentence representationscited by 4
  2. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compressioncited by 4
  3. 2025Geometric Metrics and LLMs: What They Measure and When They Workcited by 4
  4. 2025LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysiscited by 4
  5. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3
  6. 2024Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scorescited by 3
  7. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  8. 2025Toward a Flexible Framework for Linear Representation Hypothesis Using Maximum Likelihood Estimationcited by 3
  9. 2026Suppressing Final Layer Hidden State Jumps in Transformer Pretrainingcited by 3
  10. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  11. 2026Uncovering the Latent Potential of Deep Intermediate Representationscited by 3
  12. 2025Towards Lossless Token Pruning in Late-Interaction Retrieval Modelscited by 2
  13. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  14. 2026Language-Switching Triggers Take a Latent Detour Through Language Modelscited by 2
  15. 2026Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learningcited by 2
  16. 2026Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibilitycited by 2
  17. 2024Norm of Mean Contextualized Embeddings Determines their Variancecited by 1
  18. 2025Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generationcited by 1
  19. 2025Assessing the Impact of Anisotropy in Neural Representations of Speech: A Case Study on Keyword Spottingcited by 1
  20. 2025Binary Autoencoder for Mechanistic Interpretability of Large Language Modelscited by 1
  21. 2025Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEBcited by 1
  22. 2026Singular Vectors of Attention Heads Align with Featurescited by 1
  23. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directionscited by 1
  24. 2026Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerabilitycited by 1
  25. 2026Improving Relative Representations with Learned Anchors and Whitened Inner Productscited by 1
  26. 2026Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspectivecited by 1
  27. 2026Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMscited by 1
  28. 2026Sequential Adapter Stacking for Cross-Lingual Low-Resource ASRcited by 1
  29. 2024How much do contextualized representations encode long-range context?cited by 0
  30. 2025False Fixed Points: Kantian Feedback, Stable Miscalibration, and Representational Compression in LLMscited by 0
  31. 2026Capacity Constraints and the Multilingual Penalty for Lexical Disambiguationcited by 0
  32. 2026Representation Collapse in Machine Translation Through the Lens of Angular Dispersioncited by 0
  33. 2026Data-Aware Random Feature Kernel for Transformerscited by 0
  34. 2026GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMscited by 0
  35. 2026On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stabilitycited by 0
  36. 2026From Tokens to Concepts: Leveraging SAE for SPLADEcited by 0
  37. 2026Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbationscited by 0
  38. 2026The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifierscited by 0

50 most cited of 62 All 62 (JSON)