# A Fisher's exact test justification of the TF-IDF term-weighting scheme

arXiv: https://arxiv.org/abs/2507.15742 · HTML: https://mlsanity.com/paper/2507.15742/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 47 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 5 of 5)

- [A Survey on Text Classification: From Shallow to Deep Learning](https://mlsanity.com/paper/2008.00364/) (2020)
- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [The hypergeometric test performs comparably to TF-IDF on standard text analysis tasks](https://mlsanity.com/paper/2002.11844/) (2020)
- [Text classification in shipping industry using unsupervised models and Transformer based supervised models](https://mlsanity.com/paper/2212.12407/) (2022)
- [Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Text](https://mlsanity.com/paper/2303.17786/) (2023)

## Built on by (most cited first, 1 of 1)

- [Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness](https://mlsanity.com/paper/2604.00672/) (2026)

## Abstract (arXiv)

Term frequency-inverse document frequency, or TF-IDF for short, is arguably the most celebrated mathematical expression in the history of information retrieval. Conceived as a simple heuristic quantifying the extent to which a given term's occurrences are concentrated in any one given document out of many, TF-IDF and its many variants are routinely used as term-weighting schemes in diverse text analysis applications. There is a growing body of scholarship dedicated to placing TF-IDF on a sound theoretical foundation. Building on that tradition, this paper justifies the use of TF-IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective. We show that the common TF-IDF variant TF-ICF is, under mild regularity conditions, closely related to the negative logarithm of the p-value from a one-tailed version of Fisher's exact test of statistical significance. As a corollary, we establish a connection between TF-IDF and the said negative log-transformed p-value under certain idealized assumptions. We further demonstrate, as a limiting case, that this same quantity converges to TF-IDF in the limit of an infinitely large document collection. The Fisher's exact test justification of TF-IDF equips the working statistician with a ready explanation of the term-weighting scheme's long-established effectiveness.