# Exponential Family Embeddings

arXiv: https://arxiv.org/abs/1608.00778 · HTML: https://mlsanity.com/paper/1608.00778/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2016.

## Built on by (most cited first, 13 of 13)

- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Dynamic Language Models for Continuously Evolving Content](https://mlsanity.com/paper/2106.06297/) (2021)
- [Learning Rich Rankings](https://mlsanity.com/paper/2312.15081/) (2023)
- [Measuring diachronic sense change: new models and Monte Carlo methods for Bayesian inference](https://mlsanity.com/paper/2105.00819/) (2021)
- [Pseudo-Euclidean Attract-Repel Embeddings for Undirected Graphs](https://mlsanity.com/paper/2106.09671/) (2021)
- [Probabilistic Embeddings with Laplacian Graph Priors](https://mlsanity.com/paper/2204.01846/) (2022)
- [SHORING: Design Provable Conditional High-Order Interaction Network via Symbolic Testing](https://mlsanity.com/paper/2107.01326/) (2021)
- [Beer2Vec : Extracting Flavors from Reviews for Thirst-Quenching Recommandations](https://mlsanity.com/paper/2208.04223/) (2022)
- [Domain-Specific Word Embeddings with Structure Prediction](https://mlsanity.com/paper/2210.04962/) (2022)
- [WIKITIDE: A Wikipedia-Based Timestamped Definition Pairs Dataset](https://mlsanity.com/paper/2308.03582/) (2023)
- [Bypassing Skip-Gram Negative Sampling: Dimension Regularization as a More Efficient Alternative for Graph Embeddings](https://mlsanity.com/paper/2405.00172/) (2024)
- [Posterior Sampling of Probabilistic Word Embeddings](https://mlsanity.com/paper/2508.02337/) (2025)

## Abstract (arXiv)

Word embeddings are a powerful approach for capturing semantic similarity among terms in a vocabulary. In this paper, we develop exponential family embeddings, a class of methods that extends the idea of word embeddings to other types of high-dimensional data. As examples, we studied neural data with real-valued observations, count data from a market basket analysis, and ratings data from a movie recommendation system. The main idea is to model each observation conditioned on a set of other observations. This set is called the context, and the way the context is defined is a modeling choice that depends on the problem. In language the context is the surrounding words; in neuroscience the context is close-by neurons; in market basket data the context is other items in the shopping cart. Each type of embedding model defines the context, the exponential family of conditional distributions, and how the latent embedding vectors are shared across data. We infer the embeddings with a scalable algorithm based on stochastic gradient descent. On all three applications - neural activity of zebrafish, users' shopping behavior, and movie ratings - we found exponential family embedding models to be more effective than other types of dimension reduction. They better reconstruct held-out data and find interesting qualitative structure.