# Second-order Co-occurrence Sensitivity of Skip-Gram with Negative Sampling

arXiv: https://arxiv.org/abs/1906.02479 · HTML: https://mlsanity.com/paper/1906.02479/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 2 of 2)

- [Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection](https://mlsanity.com/paper/2101.09368/) (2021)
- [On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach](https://mlsanity.com/paper/2104.06474/) (2021)

## Abstract (arXiv)

We simulate first- and second-order context overlap and show that Skip-Gram with Negative Sampling is similar to Singular Value Decomposition in capturing second-order co-occurrence information, while Pointwise Mutual Information is agnostic to it. We support the results with an empirical study finding that the models react differently when provided with additional second-order information. Our findings reveal a basic property of Skip-Gram with Negative Sampling and point towards an explanation of its success on a variety of tasks.