# Posterior Sampling of Probabilistic Word Embeddings

arXiv: https://arxiv.org/abs/2508.02337 · HTML: https://mlsanity.com/paper/2508.02337/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 12 of 12)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [Frequentist Consistency of Variational Bayes](https://mlsanity.com/paper/1705.03439/) (2017)
- [Dynamic Word Embeddings](https://mlsanity.com/paper/1702.08359/) (2017)
- [Exponential Family Embeddings](https://mlsanity.com/paper/1608.00778/) (2016)
- [Dynamic Bernoulli Embeddings for Language Evolution](https://mlsanity.com/paper/1703.08052/) (2017)
- [Structured Embedding Models for Grouped Data](https://mlsanity.com/paper/1709.10367/) (2017)
- [Interpretable Word Embeddings via Informative Priors](https://mlsanity.com/paper/1909.01459/) (2019)
- [Statistical Uncertainty in Word Embeddings: GloVe-V](https://mlsanity.com/paper/2406.12165/) (2024)
- [Revisiting Skip-Gram Negative Sampling Model with Rectification](https://mlsanity.com/paper/1804.00306/) (2018)
- [Probabilistic Embeddings with Laplacian Graph Priors](https://mlsanity.com/paper/2204.01846/) (2022)

## Abstract (arXiv)

Quantifying uncertainty in word embeddings is crucial for reliable inference from textual data. However, existing Bayesian methods such as Hamiltonian Monte Carlo (HMC) and mean-field variational inference (MFVI) are either computationally infeasible for large data or rely on restrictive assumptions. We propose a scalable Gibbs sampler using Polya-Gamma augmentation as well as Laplace approximation and compare them with MFVI and HMC for word embeddings. In addition, we address non-identifiability in word embeddings. Our Gibbs sampler and HMC correctly estimate uncertainties, while MFVI does not, and Laplace approximation only does so on large sample sizes, as expected. Applying the Gibbs sampler to the US Congress and the Movielens datasets, we demonstrate the feasibility on larger real data. Finally, as a result of having draws from the full posterior, we show that the posterior mean of word embeddings improves over maximum a posteriori (MAP) estimates in terms of hold-out likelihood, especially for smaller sampling sizes, further strengthening the need for posterior sampling of word embeddings.