# On the Curious Case of ℓ2 norm of Sense Embeddings

arXiv: https://arxiv.org/abs/2210.14815 · HTML: https://mlsanity.com/paper/2210.14815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization](https://mlsanity.com/paper/2010.06478/) (2020)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings](https://mlsanity.com/paper/2203.07523/) (2022)
- [LIAAD at SemDeep-5 Challenge: Word-in-Context (WiC)](https://mlsanity.com/paper/1906.10002/) (2019)
- [You Shall Know the Most Frequent Sense by the Company it Keeps](https://mlsanity.com/paper/1808.06729/) (2018)

## Abstract (arXiv)

We show that the ℓ2 norm of a static sense embedding encodes information related to the frequency of that sense in the training corpus used to learn the sense embeddings. This finding can be seen as an extension of a previously known relationship for word embeddings to sense embeddings. Our experimental results show that, in spite of its simplicity, the ℓ2 norm of sense embeddings is a surprisingly effective feature for several word sense related tasks such as (a) most frequent sense prediction, (b) Word-in-Context (WiC), and (c) Word Sense Disambiguation (WSD). In particular, by simply including the ℓ2 norm of a sense embedding as a feature in a classifier, we show that we can improve WiC and WSD methods that use static sense embeddings.