# The Influence of Down-Sampling Strategies on SVD Word Embedding Stability

arXiv: https://arxiv.org/abs/1808.06810 · HTML: https://mlsanity.com/paper/1808.06810/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2018.

## Built on by (most cited first, 4 of 4)

- [Assessing the Reliability of Word Embedding Gender Bias Measures](https://mlsanity.com/paper/2109.04732/) (2021)
- [Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures](https://mlsanity.com/paper/2206.04330/) (2022)
- [Variation and Instability in Dialect-Based Embedding Spaces](https://mlsanity.com/paper/2303.14963/) (2023)
- [TransDrift: Modeling Word-Embedding Drift using Transformer](https://mlsanity.com/paper/2206.08081/) (2022)

## Abstract (arXiv)

The stability of word embedding algorithms, i.e., the consistency of the word representations they reveal when trained repeatedly on the same data set, has recently raised concerns. We here compare word embedding algorithms on three corpora of different sizes, and evaluate both their stability and accuracy. We find strong evidence that down-sampling strategies (used as part of their training procedures) are particularly influential for the stability of SVDPPMI-type embeddings. This finding seems to explain diverging reports on their stability and lead us to a simple modification which provides superior stability as well as accuracy on par with skip-gram embeddings.