# Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

arXiv: https://arxiv.org/abs/1504.06654 · HTML: https://mlsanity.com/paper/1504.06654/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2015.

## Built on by (most cited first, 30 of 35)

- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses](https://mlsanity.com/paper/2104.14694/) (2021)
- [Probabilistic Representations for Video Contrastive Learning](https://mlsanity.com/paper/2204.03946/) (2022)
- [Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings](https://mlsanity.com/paper/2203.07523/) (2022)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [Robust Multimodal Learning via Representation Decoupling](https://mlsanity.com/paper/2407.04458/) (2024)
- [Driving Context into Text-to-Text Privatization](https://mlsanity.com/paper/2306.01457/) (2023)
- [RICA2: Rubric-Informed, Calibrated Assessment of Actions](https://mlsanity.com/paper/2408.02138/) (2024)
- [Word-specific tonal realizations in Mandarin](https://mlsanity.com/paper/2405.07006/) (2024)
- [Can Word Sense Distribution Detect Semantic Changes of Words?](https://mlsanity.com/paper/2310.10400/) (2023)
- [Multi-sense embeddings through a word sense disambiguation process](https://mlsanity.com/paper/2101.08700/) (2021)
- [A Survey on Word Meta-Embedding Learning](https://mlsanity.com/paper/2204.11660/) (2022)
- [Introspective Deep Metric Learning](https://mlsanity.com/paper/2309.09982/) (2023)
- [Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy](https://mlsanity.com/paper/2110.02204/) (2021)
- [Quantum Mathematics in Artificial Intelligence](https://mlsanity.com/paper/2101.04255/) (2021)
- [PolyLM: Learning about Polysemy through Language Modeling](https://mlsanity.com/paper/2101.10448/) (2021)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings](https://mlsanity.com/paper/2204.12386/) (2022)
- [Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings](https://mlsanity.com/paper/2103.06459/) (2021)
- [Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications](https://mlsanity.com/paper/2103.15330/) (2021)
- [Multi-Sense Embeddings for Language Models and Knowledge Distillation](https://mlsanity.com/paper/2504.06036/) (2025)
- [A Survey On Neural Word Embeddings](https://mlsanity.com/paper/2110.01804/) (2021)
- [Discovering Fine-Grained Semantics in Knowledge Graph Relations](https://mlsanity.com/paper/2202.08917/) (2022)
- [CKG: Dynamic Representation Based on Context and Knowledge Graph](https://mlsanity.com/paper/2212.04909/) (2022)
- [EDS-MEMBED: Multi-sense embeddings based on enhanced distributional semantic structures via a graph walk over word senses](https://mlsanity.com/paper/2103.00232/) (2021)
- [Multi-facet Universal Schema](https://mlsanity.com/paper/2103.15339/) (2021)
- [A Comprehensive Empirical Evaluation of Existing Word Embedding Approaches](https://mlsanity.com/paper/2303.07196/) (2023)
- [Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings](https://mlsanity.com/paper/2305.19092/) (2023)
- [Sense representations for Portuguese: experiments with sense embeddings and deep neural language models](https://mlsanity.com/paper/2109.00025/) (2021)
- [A Comparison of Approaches for Imbalanced Classification Problems in the Context of Retrieving Relevant Documents for an Analysis](https://mlsanity.com/paper/2205.01600/) (2022)

## Abstract (arXiv)

There is rising interest in vector-space word embeddings and their use in NLP, especially given recent methods for their fast estimation at very large scale. Nearly all this work, however, assumes a single vector per word type ignoring polysemy and thus jeopardizing their usefulness for downstream tasks. We present an extension to the Skip-gram model that efficiently learns multiple embeddings per word type. It differs from recent related work by jointly performing word sense discrimination and embedding learning, by non-parametrically estimating the number of senses per word type, and by its efficiency and scalability. We present new state-of-the-art results in the word similarity in context task and demonstrate its scalability by training with one machine on a corpus of nearly 1 billion tokens in less than 6 hours.