# Using k-way Co-occurrences for Learning Word Embeddings

arXiv: https://arxiv.org/abs/1709.01199 · HTML: https://mlsanity.com/paper/1709.01199/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2017.

## Built on by (most cited first, 5 of 5)

- [A Survey on Word Meta-Embedding Learning](https://mlsanity.com/paper/2204.11660/) (2022)
- [Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discounting](https://mlsanity.com/paper/2305.10610/) (2023)
- [A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings](https://mlsanity.com/paper/2309.10551/) (2023)
- [RelWalk A Latent Variable Model Approach to Knowledge Graph Embedding](https://mlsanity.com/paper/2101.10070/) (2021)
- [Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspective](https://mlsanity.com/paper/2508.08140/) (2025)

## Abstract (arXiv)

Co-occurrences between two words provide useful insights into the semantics of those words. Consequently, numerous prior work on word embedding learning have used co-occurrences between two words as the training signal for learning word embeddings. However, in natural language texts it is common for multiple words to be related and co-occurring in the same context. We extend the notion of co-occurrences to cover k(≥!!2)-way co-occurrences among a set of k-words. Specifically, we prove a theoretical relationship between the joint probability of k(≥!!2) words, and the sum of ℓ2 norms of their embeddings. Next, we propose a learning objective motivated by our theoretical result that utilises k-way co-occurrences for learning word embeddings. Our experimental results show that the derived theoretical relationship does indeed hold empirically, and despite data sparsity, for some smaller k values, k-way embeddings perform comparably or better than 2-way embeddings in a range of tasks.