# Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics

arXiv: https://arxiv.org/abs/1908.06941 · HTML: https://mlsanity.com/paper/1908.06941/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 4 of 4)

- [Linguistic Dependencies and Statistical Dependence](https://mlsanity.com/paper/2104.08685/) (2021)
- [Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks](https://mlsanity.com/paper/2306.06472/) (2023)
- [EigenNoise: A Contrastive Prior to Warm-Start Representations](https://mlsanity.com/paper/2205.04376/) (2022)
- [The "Small World of Words" German Free-Association Norms](https://mlsanity.com/paper/2604.19620/) (2026)

## Abstract (arXiv)

In distributional semantics, the pointwise mutual information (mathitPMI) weighting of the cooccurrence matrix performs far better than raw counts. There is, however, an issue with unobserved pair cooccurrences as mathitPMI goes to negative infinity. This problem is aggravated by unreliable statistics from finite corpora which lead to a large number of such pairs. A common practice is to clip negative mathitPMI (mathittexttt- PMI) at 0, also known as Positive mathitPMI (mathitPPMI). In this paper, we investigate alternative ways of dealing with mathittexttt- PMI and, more importantly, study the role that negative information plays in the performance of a low-rank, weighted factorization of different mathitPMI matrices. Using various semantic and syntactic tasks as probes into models which use either negative or positive mathitPMI (or both), we find that most of the encoded semantics and syntax come from positive mathitPMI, in contrast to mathittexttt- PMI which contributes almost exclusively syntactic information. Our findings deepen our understanding of distributional semantics, while also introducing novel PMI variants and grounding the popular PPMI measure.