# Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness

arXiv: https://arxiv.org/abs/2604.00672 · HTML: https://mlsanity.com/paper/2604.00672/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 37 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 5 of 5)

- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [Modelling Word Burstiness in Natural Language: A Generalised Polya Process for Document Language Models in Information Retrieval](https://mlsanity.com/paper/1708.06011/) (2017)
- [The hypergeometric test performs comparably to TF-IDF on standard text analysis tasks](https://mlsanity.com/paper/2002.11844/) (2020)
- [A Polya Urn Document Language Model for Improved Information Retrieval](https://mlsanity.com/paper/1502.00804/) (2015)
- [A Fisher's exact test justification of the TF-IDF term-weighting scheme](https://mlsanity.com/paper/2507.15742/) (2025)

## Abstract (arXiv)

TF-IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks. This paper provides insights into TF-IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.