# Factors Influencing the Surprising Instability of Word Embeddings

arXiv: https://arxiv.org/abs/1804.09692 · HTML: https://mlsanity.com/paper/1804.09692/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2018.

## Built on by (most cited first, 30 of 33)

- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models](https://mlsanity.com/paper/2110.00672/) (2021)
- [Shared Global and Local Geometry of Language Model Embeddings](https://mlsanity.com/paper/2503.21073/) (2025)
- [Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics](https://mlsanity.com/paper/2206.03390/) (2022)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora](https://mlsanity.com/paper/2112.14330/) (2021)
- [Frequency-based Distortions in Contextualized Word Embeddings](https://mlsanity.com/paper/2104.08465/) (2021)
- [Statistically significant detection of semantic shifts using contextual word embeddings](https://mlsanity.com/paper/2104.03776/) (2021)
- [Assessing the Reliability of Word Embedding Gender Bias Measures](https://mlsanity.com/paper/2109.04732/) (2021)
- [A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?](https://mlsanity.com/paper/2305.12920/) (2023)
- [Observatory: Characterizing Embeddings of Relational Tables](https://mlsanity.com/paper/2310.07736/) (2023)
- [Reducing Retraining by Recycling Parameter-Efficient Prompts](https://mlsanity.com/paper/2208.05577/) (2022)
- [Robust Concept Erasure via Kernelized Rate-Distortion Maximization](https://mlsanity.com/paper/2312.00194/) (2023)
- [Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions](https://mlsanity.com/paper/2202.09166/) (2022)
- [Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems](https://mlsanity.com/paper/2108.05053/) (2021)
- [Evolving linguistic divergence on polarizing social media](https://mlsanity.com/paper/2309.01659/) (2023)
- [Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discounting](https://mlsanity.com/paper/2305.10610/) (2023)
- [Statistical Uncertainty in Word Embeddings: GloVe-V](https://mlsanity.com/paper/2406.12165/) (2024)
- [Are Word Embedding Methods Stable and Should We Care About It?](https://mlsanity.com/paper/2104.08433/) (2021)
- [A Greek Parliament Proceedings Dataset for Computational Linguistics and Political Analysis](https://mlsanity.com/paper/2210.12883/) (2022)
- [Analysis and Prediction of NLP Models Via Task Embeddings](https://mlsanity.com/paper/2112.05647/) (2021)
- [Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures](https://mlsanity.com/paper/2206.04330/) (2022)
- [Variation and Instability in Dialect-Based Embedding Spaces](https://mlsanity.com/paper/2303.14963/) (2023)
- [Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space](https://mlsanity.com/paper/2304.14333/) (2023)
- [TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora](https://mlsanity.com/paper/2103.11029/) (2021)
- [TransDrift: Modeling Word-Embedding Drift using Transformer](https://mlsanity.com/paper/2206.08081/) (2022)
- [Probing Taxonomic and Thematic Embeddings for Taxonomic Information](https://mlsanity.com/paper/2301.10656/) (2023)
- [The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instruments](https://mlsanity.com/paper/2603.10130/) (2026)
- [An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications](https://mlsanity.com/paper/2204.09885/) (2022)
- [Dialectograms: Machine Learning Differences between Discursive Communities](https://mlsanity.com/paper/2302.05657/) (2023)

## Abstract (arXiv)

Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this paper, we consider one aspect of embedding spaces, namely their stability. We show that even relatively high frequency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the effects of stability on downstream tasks.