# Non-Zipfian Distribution of Stopwords or Function Words and Subset Selection Models

arXiv: https://arxiv.org/abs/2603.04691 · HTML: https://mlsanity.com/paper/2603.04691/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 52 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 5 of 5)

- [Stopwords in Technical Language Processing](https://mlsanity.com/paper/2006.02633/) (2020)
- [Authorship Attribution through Function Word Adjacency Networks](https://mlsanity.com/paper/1406.4469/) (2014)
- [Beta Rank Function: A Smooth Double-Pareto-Like Distribution](https://mlsanity.com/paper/1910.05364/) (2019)
- [Human mobility patterns in Mexico City and their links with socioeconomic variables during the COVID-19 pandemic](https://mlsanity.com/paper/2208.06456/) (2022)
- [Quadratic Term Correction on Heaps' Law](https://mlsanity.com/paper/2511.14683/) (2025)

## Abstract (arXiv)

Stopwords and function words are relatively less informative for the content of a language and more often play a structural role in a sentence. Stopwords are ubiquitous words and may contain verbs, adjectives and adverbs. On the other hand, function words are strictly prepositions, conjunctions, pronouns, determiners, qualifiers, articles, interrogatives, and a limited number of auxiliary verbs. In contrast to the well known Zipf's law for rank-frequency plot for all words, the rank-frequency plots for stopwords or function words are best fitted by the Beta Rank Function (BRF). On the other hand, the rank-frequency plots of non-stopwords or non-function-words also deviate from the Zipf's law, but are better described by a quadratic function of log-token-count over log-rank than by BRF. Based on the observed rank of stopwords or function words in the full word list, we propose a stopword/function word/subset selection model that the probability for being selected, as a function of the word's rank r, is a decreasing Hill's function (1/(1+(r/rmid)γ)); whereas the probability for not being selected is the standard Hill's function (1/(1+(rmid/r)γ)). We validate this selection probability model by a direct estimation from an independent collection of texts. We also show analytically that this model leads to a BRF rank-frequency distribution for stopwords or function words when the original full word list follows the Zipf's law, as well as explaining the quadratic fitting function for the non-stopwords or non-function-words. A corollary of these results is that Zipf's law is not expected to be true for telegraphic speech in early childhood language learners or in agrammatism patients.