mlsanityData to 7 Oct 2026

Factors Influencing the Surprising Instability of Word Embeddings

arXiv 1804.09692 ↗First posted 25 Apr 2018cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this paper, we consider one aspect of embedding spaces, namely their stability. We show that even relatively high frequency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the effects of stability on downstream tasks.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 10; 2024: 1; 2025: 1; 2026: 2.

31 papers.

Sort:

  1. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  2. 2021Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Modelscited by 39
  3. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  4. 2022Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semanticscited by 24
  5. 2021Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corporacited by 21
  6. 2023A Survey on Contextualised Semantic Shift Detectioncited by 21
  7. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 19
  8. 2021Statistically significant detection of semantic shifts using contextual word embeddingscited by 14
  9. 2021Assessing the Reliability of Word Embedding Gender Bias Measurescited by 14
  10. 2023A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?cited by 14
  11. 2023Observatory: Characterizing Embeddings of Relational Tablescited by 12
  12. 2022Reducing Retraining by Recycling Parameter-Efficient Promptscited by 11
19 more
  1. 2023Robust Concept Erasure via Kernelized Rate-Distortion Maximizationcited by 10
  2. 2022Evaluating the Construct Validity of Text Embeddings with Application to Survey Questionscited by 7
  3. 2021Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystemscited by 6
  4. 2023Evolving linguistic divergence on polarizing social mediacited by 6
  5. 2023Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discountingcited by 5
  6. 2021Are Word Embedding Methods Stable and Should We Care About It?cited by 4
  7. 2022A Greek Parliament Proceedings Dataset for Computational Linguistics and Political Analysiscited by 4
  8. 2024Statistical Uncertainty in Word Embeddings: GloVe-Vcited by 4
  9. 2021Analysis and Prediction of NLP Models Via Task Embeddingscited by 3
  10. 2022Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measurescited by 3
  11. 2023Variation and Instability in Dialect-Based Embedding Spacescited by 3
  12. 2023Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Spacecited by 3
  13. 2021TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corporacited by 1
  14. 2022TransDrift: Modeling Word-Embedding Drift using Transformercited by 1
  15. 2023Probing Taxonomic and Thematic Embeddings for Taxonomic Informationcited by 1
  16. 2026The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instrumentscited by 1
  17. 2022An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applicationscited by 0
  18. 2023Dialectograms: Machine Learning Differences between Discursive Communitiescited by 0
  19. 2026The Impact of Dimensionality on the Stability of Node Embeddingscited by 0