Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics

In distributional semantics, the pointwise mutual information (mathitPMI) weighting of the cooccurrence matrix performs far better than raw counts. There is, however, an issue with unobserved pair cooccurrences as mathitPMI goes to negative infinity. This problem is aggravated by unreliable statistics from finite corpora which lead to a large number of such pairs. A common practice is to clip negative mathitPMI (mathittexttt- PMI) at 0, also known as Positive mathitPMI (mathitPPMI). In this paper, we investigate alternative ways of dealing with mathittexttt- PMI and, more importantly, study the role that negative information plays in the performance of a low-rank, weighted factorization of different mathitPMI matrices. Using various semantic and syntactic tasks as probes into models which use either negative or positive mathitPMI (or both), we find that most of the encoded semantics and syntax come from positive mathitPMI, in contrast to mathittexttt- PMI which contributes almost exclusively syntactic information. Our findings deepen our understanding of distributional semantics, while also introducing novel PMI variants and grounding the popular PPMI measure.