# Unigram-Normalized Perplexity as a Language Model Performance Measure with Different Vocabulary Sizes

arXiv: https://arxiv.org/abs/2011.13220 · HTML: https://mlsanity.com/paper/2011.13220/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2020.

## Built on by (most cited first, 6 of 6)

- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://mlsanity.com/paper/2403.15796/) (2024)
- [How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns](https://mlsanity.com/paper/2204.10281/) (2022)
- [Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries](https://mlsanity.com/paper/2506.01535/) (2025)
- [TextGram: Towards a better domain-adaptive pretraining](https://mlsanity.com/paper/2404.18228/) (2024)
- [Classifier Language Models: Unifying Sparse Finetuning and Adaptive Tokenization for Specialized Classification Tasks](https://mlsanity.com/paper/2508.08635/) (2025)

## Abstract (arXiv)

Although Perplexity is a widely used performance metric for language models, the values are highly dependent upon the number of words in the corpus and is useful to compare performance of the same corpus only. In this paper, we propose a new metric that can be used to evaluate language model performance with different vocabulary sizes. The proposed unigram-normalized Perplexity actually presents the performance improvement of the language models from that of simple unigram model, and is robust on the vocabulary size. Both theoretical analysis and computational experiments are reported.