mlsanityData to 7 Oct 2026

Isotropy, Clusters, and Classifiers

arXiv 2402.03191 ↗First posted 5 Feb 2024cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Whether embedding spaces use all their dimensions equally, i.e., whether they are isotropic, has been a recent subject of discussion. Evidence has been accrued both for and against enforcing isotropy in embedding spaces. In the present paper, we stress that isotropy imposes requirements on the embedding space that are not compatible with the presence of clusters -- which also negatively impacts linear classification objectives. We demonstrate this fact both mathematically and empirically and use it to shed light on previous results from the literature.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 44 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

23 of 44 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  9. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  10. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  11. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  12. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
11 more
  1. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  2. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  3. 2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
  4. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  5. 2022How to Dissect a Muppet: The Structure of Transformer Embedding Spacescited by 19
  6. 2022Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddingscited by 18
  7. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  8. 2022Isotropic Representation Can Improve Dense Retrievalcited by 8
  9. 2023Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictionscited by 6
  10. 2023Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representationscited by 4
  11. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Meaning at the Planck scale? Contextualized word embeddings for doing history, philosophy, and sociology of sciencecited by 3
  2. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  3. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  4. 2025Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEBcited by 1