# The hypergeometric test performs comparably to TF-IDF on standard text analysis tasks

arXiv: https://arxiv.org/abs/2002.11844 · HTML: https://mlsanity.com/paper/2002.11844/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 2 of 2)

- [A Fisher's exact test justification of the TF-IDF term-weighting scheme](https://mlsanity.com/paper/2507.15742/) (2025)
- [Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness](https://mlsanity.com/paper/2604.00672/) (2026)

## Abstract (arXiv)

Term frequency-inverse document frequency, or TF-IDF for short, and its many variants form a class of term weighting functions the members of which are widely used in text analysis applications. While TF-IDF was originally proposed as a heuristic, theoretical justifications grounded in information theory, probability, and the divergence from randomness paradigm have been advanced. In this work, we present an empirical study showing that TF-IDF corresponds very nearly with the hypergeometric test of statistical significance on selected real-data document retrieval, summarization, and classification tasks. These findings suggest that a fundamental mathematical connection between TF-IDF and the negative logarithm of the hypergeometric test P-value (i.e., a hypergeometric distribution tail probability) remains to be elucidated. We advance the empirical analyses herein as a first step toward explaining the long-standing effectiveness of TF-IDF from a statistical significance testing lens. It is our aspiration that these results will open the door to the systematic evaluation of significance testing derived term weighting functions in text analysis applications.