# Norm of Word Embedding Encodes Information Gain

arXiv: https://arxiv.org/abs/2212.09663 · HTML: https://mlsanity.com/paper/2212.09663/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 80 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications](https://mlsanity.com/paper/1704.02853/) (2017)
- [Analogies Explained: Towards Understanding Word Embeddings](https://mlsanity.com/paper/1901.09813/) (2019)
- [Word Rotator's Distance](https://mlsanity.com/paper/2004.15003/) (2020)
- [Measuring Word Significance using Distributed Representations of Words](https://mlsanity.com/paper/1508.02297/) (2015)
- [A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors](https://mlsanity.com/paper/1805.05388/) (2018)
- [Hypernyms under Siege: Linguistically-motivated Artillery for Hypernymy Detection](https://mlsanity.com/paper/1612.04460/) (2016)
- [What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris](https://mlsanity.com/paper/2205.07750/) (2022)
- [Keyphrase Cloud Generation of Broadcast News](https://mlsanity.com/paper/1306.4606/) (2013)
- [How much does a word weigh? Weighting word embeddings for word sense induction](https://mlsanity.com/paper/1805.09209/) (2018)
- [More than just Frequency? Demasking Unsupervised Hypernymy Prediction Methods](https://mlsanity.com/paper/2106.00055/) (2021)

## Built on by (most cited first, 21 of 21)

- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [Semantics at an Angle: When Cosine Similarity Works Until It Doesn't](https://mlsanity.com/paper/2504.16318/) (2025)
- [Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words](https://mlsanity.com/paper/2310.15921/) (2023)
- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference](https://mlsanity.com/paper/2501.15754/) (2025)
- [Learning is Forgetting: LLM Training As Lossy Compression](https://mlsanity.com/paper/2604.07569/) (2026)
- [NormXLogit: The Head-on-Top Never Lies](https://mlsanity.com/paper/2411.16252/) (2024)
- [Zipfian Whitening](https://mlsanity.com/paper/2411.00680/) (2024)
- [Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics](https://mlsanity.com/paper/2604.08764/) (2026)
- [Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation](https://mlsanity.com/paper/2406.08482/) (2024)
- [Norm of Mean Contextualized Embeddings Determines their Variance](https://mlsanity.com/paper/2409.11253/) (2024)
- [OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models](https://mlsanity.com/paper/2607.23193/) (2026)
- [CLIP-like Model as a Foundational Density Ratio Estimator](https://mlsanity.com/paper/2506.22881/) (2025)
- [SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora](https://mlsanity.com/paper/2602.10908/) (2026)
- [On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode](https://mlsanity.com/paper/2603.13911/) (2026)
- [What makes a word hard to learn? Modeling L1 influence on English vocabulary difficulty](https://mlsanity.com/paper/2605.12281/) (2026)
- [Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference](https://mlsanity.com/paper/2606.16461/) (2026)
- [PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning](https://mlsanity.com/paper/2606.21139/) (2026)
- [Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms](https://mlsanity.com/paper/2606.30625/) (2026)
- [Rethinking Contextualization by Reinterpreting Attention Head Channels](https://mlsanity.com/paper/2609.33851/) (2026)
- [FineSID: Scalable and Efficient Semantic Identifier Learning for Generative Recommendation](https://mlsanity.com/paper/2609.36670/) (2026)

## Abstract (arXiv)

Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution. Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contextualized word embeddings in language models or any neural networks with the softmax output layer. We also demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of the informativeness of a word in tasks such as keyword extraction, proper-noun discrimination, and hypernym discrimination.