mlsanityData to 14 Oct 2026

IsoScore: Measuring the Uniformity of Embedding Space Utilization

arXiv 2108.07344 ↗First posted 16 Aug 2021cs.CL cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

The recent success of distributed word representations has led to an increased interest in analyzing the properties of their spatial distribution. Several studies have suggested that contextualized word embedding models do not isotropically project tokens into vector space. However, current methods designed to measure isotropy, such as average random cosine similarity and the partition score, have not been thoroughly analyzed and are not appropriate for measuring isotropy. We propose IsoScore: a novel tool that quantifies the degree to which a point cloud uniformly utilizes the ambient vector space. Using rigorously designed tests, we demonstrate that IsoScore is the only tool available in the literature that accurately measures how uniformly distributed variance is across dimensions in vector space. Additionally, we use IsoScore to challenge a number of recent conclusions in the NLP literature that have been derived using brittle metrics of isotropy. We caution future studies from using existing tools to measure isotropy in contextualized embedding space as resulting conclusions will be misleading or altogether inaccurate.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 13 of its 30 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

13 of 30 references.

References · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  5. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  6. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  7. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  8. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  9. 2018FRAGE: Frequency-Agnostic Word Representationcited by 47
  10. 2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
  11. 2019What do you mean, BERT? Assessing BERT as a Distributional Semantics Modelcited by 14
  12. 2020IsoBN: Fine-Tuning BERT with Isotropic Batch Normalizationcited by 12
1 more
  1. 2015A Latent Variable Model Approach to PMI-based Word Embeddingscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 13; 2025: 8; 2026: 22.

53 papers.

Sort:

By area:

  1. 2024Hyperbolic Fine-Tuning for Large Language Modelscited by 27
  2. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 21
  3. 2023Byte Pair Encoding for Symbolic Musiccited by 19
  4. 2024Robust AI-Generated Text Detection by Restricted Embeddingscited by 18
  5. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  6. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  7. 2023Exploring Distributional Shifts in Large Language Models for Code Analysiscited by 14
  8. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  9. 2023Stable Anisotropic Regularizationcited by 12
  10. 2024Dissecting Query-Key Interaction in Vision Transformerscited by 12
  11. 2025Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddingscited by 11
  12. 2024Can Large Language Models Understand Molecules?cited by 10
38 more
  1. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 8
  2. 2022IsoVec: Controlling the Relative Isomorphism of Word Embedding Spacescited by 7
  3. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  4. 2022The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfercited by 5
  5. 2024Improving Interpretability and Robustness for the Detection of AI-Generated Imagescited by 5
  6. 2025From Topology to Retrieval: Decoding Embedding Spaces with Unified Signaturescited by 5
  7. 2024Isotropy, Clusters, and Classifierscited by 4
  8. 2025Better Embeddings with Coupled Adamcited by 4
  9. 2023Analyzing the Representational Geometry of Acoustic Word Embeddingscited by 3
  10. 2024Zipfian Whiteningcited by 3
  11. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 3
  12. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  13. 2025Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortioncited by 2
  14. 2025Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Searchcited by 2
  15. 2025Towards Understanding the Shape of Representations in Protein Language Modelscited by 2
  16. 2023Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Modelscited by 1
  17. 2024Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performancecited by 1
  18. 2024Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrievalcited by 1
  19. 2024Whitening Not Recommended for Classification Tasks in LLMscited by 1
  20. 2026Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Indexcited by 1
  21. 2023Exploring Representational Disparities Between Multilingual and Bilingual Translation Modelscited by 0
  22. 2025HomID : Benchmarking Intrinsic Dimension Estimators on Homogenous Manifolds with Anisotropic Embeddingscited by 0
  23. 2026Isotropy-Optimized Contrastive Learning for Semantic Course Recommendationcited by 0
  24. 2026Capacity Constraints and the Multilingual Penalty for Lexical Disambiguationcited by 0
  25. 2026Degradation of Feature Space in Continual Learningcited by 0
  26. 2026GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Modelscited by 0
  27. 2026Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlationscited by 0
  28. 2026On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stabilitycited by 0
  29. 2026REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuningcited by 0
  30. 2026FLARE: Task-agnostic embedding model evaluation through a normalization processcited by 0
  31. 2026Multimodal QUD: Inquisitive Questions from Scientific Figurescited by 0
  32. 2026How Many Different Outputs Can a Transformer Generate?cited by 0
  33. 2026Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddingscited by 0
  34. 2026Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Modelscited by 0
  35. 2026ChaosProbe: A Neurochaotic Lens on Frozen Transformer Input-Embedding Spacescited by 0
  36. 2026The Geometry of Low-Resource Language Representationscited by 0
  37. 2026A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddingscited by 0
  38. 2026A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failurescited by 0

50 most cited of 53 All 53 (JSON)