# Heaps' Law in GPT-Neo Large Language Model Emulated Corpora

arXiv: https://arxiv.org/abs/2311.06377 · HTML: https://mlsanity.com/paper/2311.06377/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 18 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 4 of 4)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [Heaps' law and Heaps functions in tagged texts: Evidences of their linguistic relevance](https://mlsanity.com/paper/2001.02178/) (2020)
- [Scaling laws and fluctuations in the statistics of word frequencies](https://mlsanity.com/paper/1406.4441/) (2014)

## Built on by (most cited first, 1 of 1)

- [From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis](https://mlsanity.com/paper/2512.13491/) (2025)

## Abstract (arXiv)

Heaps' law is an empirical relation in text analysis that predicts vocabulary growth as a function of corpus size. While this law has been validated in diverse human-authored text corpora, its applicability to large language model generated text remains unexplored. This study addresses this gap, focusing on the emulation of corpora using the suite of GPT-Neo large language models. To conduct our investigation, we emulated corpora of PubMed abstracts using three different parameter sizes of the GPT-Neo model. Our emulation strategy involved using the initial five words of each PubMed abstract as a prompt and instructing the model to expand the content up to the original abstract's length. Our findings indicate that the generated corpora adhere to Heaps' law. Interestingly, as the GPT-Neo model size grows, its generated vocabulary increasingly adheres to Heaps' law as as observed in human-authored text. To further improve the richness and authenticity of GPT-Neo outputs, future iterations could emphasize enhancing model size or refining the model architecture to curtail vocabulary repetition.