# A Simple Language Model based on PMI Matrix Approximations

arXiv: https://arxiv.org/abs/1707.05266 · HTML: https://mlsanity.com/paper/1707.05266/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2017.

## Built on by (most cited first, 2 of 2)

- [Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding](https://mlsanity.com/paper/2106.07250/) (2021)
- [Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding](https://mlsanity.com/paper/2407.04251/) (2024)

## Abstract (arXiv)

In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec's algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.