# Pragmatic Constraint on Distributional Semantics

arXiv: https://arxiv.org/abs/2211.11041 · HTML: https://mlsanity.com/paper/2211.11041/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 20 Nov 2022.

## Builds on (most cited first, 6 of 6)

- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Do Neural Nets Learn Statistical Laws behind Natural Language?](https://mlsanity.com/paper/1707.04848/) (2017)
- [How (Non-)Optimal is the Lexicon?](https://mlsanity.com/paper/2104.14279/) (2021)
- [The origins of Zipf's meaning-frequency law](https://mlsanity.com/paper/1801.00168/) (2017)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)

## Built on by (most cited first, 2 of 2)

- [MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers](https://mlsanity.com/paper/2602.00398/) (2026)
- [COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens](https://mlsanity.com/paper/2509.06836/) (2025)

## Abstract (arXiv)

This paper studies the limits of language models' statistical learning in the context of Zipf's law. First, we demonstrate that Zipf-law token distribution emerges irrespective of the chosen tokenization. Second, we show that Zipf distribution is characterized by two distinct groups of tokens that differ both in terms of their frequency and their semantics. Namely, the tokens that have a one-to-one correspondence with one semantic concept have different statistical properties than those with semantic ambiguity. Finally, we demonstrate how these properties interfere with statistical learning procedures motivated by distributional semantics.