# Self-Normalization Properties of Language Modeling

arXiv: https://arxiv.org/abs/1806.00913 · HTML: https://mlsanity.com/paper/1806.00913/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2018.

## Built on by (most cited first, 3 of 3)

- [Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES](https://mlsanity.com/paper/2205.00704/) (2022)
- [On Sampling-Based Training Criteria for Neural Language Modeling](https://mlsanity.com/paper/2104.10507/) (2021)
- [Self-Normalized Importance Sampling for Neural Language Modeling](https://mlsanity.com/paper/2111.06310/) (2021)

## Abstract (arXiv)

Self-normalizing discriminative models approximate the normalized probability of a class without having to compute the partition function. In the context of language modeling, this property is particularly appealing as it may significantly reduce run-times due to large word vocabularies. In this study, we provide a comprehensive investigation of language modeling self-normalization. First, we theoretically analyze the inherent self-normalization properties of Noise Contrastive Estimation (NCE) language models. Then, we compare them empirically to softmax-based approaches, which are self-normalized using explicit regularization, and suggest a hybrid model with compelling properties. Finally, we uncover a surprising negative correlation between self-normalization and perplexity across the board, as well as some regularity in the observed errors, which may potentially be used for improving self-normalization algorithms in the future.