# PolyLM: Learning about Polysemy through Language Modeling

arXiv: https://arxiv.org/abs/2101.10448 · HTML: https://mlsanity.com/paper/2101.10448/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space](https://mlsanity.com/paper/1504.06654/) (2015)
- [Word Sense Induction with Neural biLM and Symmetric Patterns](https://mlsanity.com/paper/1808.08518/) (2018)
- [Towards better substitution-based word sense induction](https://mlsanity.com/paper/1905.12598/) (2019)
- [Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation](https://mlsanity.com/paper/1905.05677/) (2019)
- [Breaking Sticks and Ambiguities with Adaptive Skip-gram](https://mlsanity.com/paper/1502.07257/) (2015)
- [Do Multi-Sense Embeddings Improve Natural Language Understanding?](https://mlsanity.com/paper/1506.01070/) (2015)
- [AutoSense Model for Word Sense Induction](https://mlsanity.com/paper/1811.09242/) (2018)
- [MUSE: Modularizing Unsupervised Sense Embeddings](https://mlsanity.com/paper/1704.04601/) (2017)
- [Intrinsic Subspace Evaluation of Word Embedding Representations](https://mlsanity.com/paper/1606.07902/) (2016)
- [Sense Embedding Learning for Word Sense Induction](https://mlsanity.com/paper/1606.05409/) (2016)

## Built on by (most cited first, 5 of 5)

- [In the LLM era, Word Sense Induction remains unsolved](https://mlsanity.com/paper/2603.11686/) (2026)
- [Multilingual Substitution-based Word Sense Induction](https://mlsanity.com/paper/2405.11086/) (2024)
- [QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting](https://mlsanity.com/paper/2508.16697/) (2025)
- [Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization](https://mlsanity.com/paper/2210.05422/) (2022)
- [No One Size Fits All: QueryBandits for Hallucination Mitigation](https://mlsanity.com/paper/2602.20332/) (2026)

## Abstract (arXiv)

To avoid the "meaning conflation deficiency" of word embeddings, a number of models have aimed to embed individual word senses. These methods at one time performed well on tasks such as word sense induction (WSI), but they have since been overtaken by task-specific techniques which exploit contextualized embeddings. However, sense embeddings and contextualization need not be mutually exclusive. We introduce PolyLM, a method which formulates the task of learning sense embeddings as a language modeling problem, allowing contextualization techniques to be applied. PolyLM is based on two underlying assumptions about word senses: firstly, that the probability of a word occurring in a given context is equal to the sum of the probabilities of its individual senses occurring; and secondly, that for a given occurrence of a word, one of its senses tends to be much more plausible in the context than the others. We evaluate PolyLM on WSI, showing that it performs considerably better than previous sense embedding techniques, and matches the current state-of-the-art specialized WSI method despite having six times fewer parameters. Code and pre-trained models are available at https://github.com/AlanAnsell/PolyLM.