mlsanityData to 7 Oct 2026

Is Anisotropy Inherent to Transformers?

arXiv 2306.07656 ↗First posted 13 Jun 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 40 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

34 of 40 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
22 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  10. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  11. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  12. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  13. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  14. 2022Visual Attention Networkcited by 210
  15. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  16. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  17. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  18. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  19. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  20. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  21. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 19
  22. 2021Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representationscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4.

5 papers.

  1. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  2. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12
  3. 2024Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scorescited by 3
  4. 2024Meaning at the Planck scale? Contextualized word embeddings for doing history, philosophy, and sociology of sciencecited by 3
  5. 2024Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrievalcited by 1