# Embedding Words as Distributions with a Bayesian Skip-gram Model

arXiv: https://arxiv.org/abs/1711.11027 · HTML: https://mlsanity.com/paper/1711.11027/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2017.

## Built on by (most cited first, 9 of 9)

- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication task](https://mlsanity.com/paper/2105.06546/) (2021)
- [The Conceptual VAE](https://mlsanity.com/paper/2203.11216/) (2022)
- [Representation Learning via Variational Bayesian Networks](https://mlsanity.com/paper/2306.16326/) (2023)
- [From Conceptual Spaces to Quantum Concepts: Formalising and Learning Structured Conceptual Models](https://mlsanity.com/paper/2401.08585/) (2023)
- [Rethinking Word Similarity: Semantic Similarity through Classification Confusion](https://mlsanity.com/paper/2502.05704/) (2025)
- [Empirical Study of Named Entity Recognition Performance Using Distribution-aware Word Embedding](https://mlsanity.com/paper/2109.01636/) (2021)
- [Word Embedding with Neural Probabilistic Prior](https://mlsanity.com/paper/2309.11824/) (2023)
- [Alcmean's: Unsupervised community detection using local Laplacian, automatic detection of the number of centers](https://mlsanity.com/paper/2606.09100/) (2026)

## Abstract (arXiv)

We introduce a method for embedding words as probability densities in a low-dimensional space. Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods, in our Bayesian model we generate it from a word-specific prior density for each occurrence of a given word. Intuitively, for each word, the prior density encodes the distribution of its potential 'meanings'. These prior densities are conceptually similar to Gaussian embeddings. Interestingly, unlike the Gaussian embeddings, we can also obtain context-specific densities: they encode uncertainty about the sense of a word given its context and correspond to posterior distributions within our model. The context-dependent densities have many potential applications: for example, we show that they can be directly used in the lexical substitution task. We describe an effective estimation method based on the variational autoencoding framework. We also demonstrate that our embeddings achieve competitive results on standard benchmarks.