# Large Scale Substitution-based Word Sense Induction

arXiv: https://arxiv.org/abs/2110.07681 · HTML: https://mlsanity.com/paper/2110.07681/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 53 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [Syntactic Search by Example](https://mlsanity.com/paper/2006.03010/) (2020)
- [AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes](https://mlsanity.com/paper/1507.01127/) (2015)
- [Word Sense Induction with Neural biLM and Symmetric Patterns](https://mlsanity.com/paper/1808.08518/) (2018)
- [Towards better substitution-based word sense induction](https://mlsanity.com/paper/1905.12598/) (2019)
- [WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context](https://mlsanity.com/paper/2004.15016/) (2020)
- [De-Conflated Semantic Representations](https://mlsanity.com/paper/1608.01961/) (2016)
- [Making Sense of Word Embeddings](https://mlsanity.com/paper/1708.03390/) (2017)
- [Embedding Words and Senses Together via Joint Knowledge-Enhanced Training](https://mlsanity.com/paper/1612.02703/) (2016)
- [AutoSense Model for Word Sense Induction](https://mlsanity.com/paper/1811.09242/) (2018)
- [Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations](https://mlsanity.com/paper/1611.01547/) (2016)

## Built on by (most cited first, 9 of 9)

- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications](https://mlsanity.com/paper/2212.09676/) (2022)
- [Substitution-based Semantic Change Detection using Contextual Embeddings](https://mlsanity.com/paper/2309.02403/) (2023)
- [To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models](https://mlsanity.com/paper/2406.20054/) (2024)
- [What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification](https://mlsanity.com/paper/2510.03541/) (2025)
- [In the LLM era, Word Sense Induction remains unsolved](https://mlsanity.com/paper/2603.11686/) (2026)
- [Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabulary](https://mlsanity.com/paper/2210.06376/) (2022)
- [Context-Aware Semantic Similarity Measurement for Unsupervised Word Sense Disambiguation](https://mlsanity.com/paper/2305.03520/) (2023)
- [Multilingual Substitution-based Word Sense Induction](https://mlsanity.com/paper/2405.11086/) (2024)

## Abstract (arXiv)

We present a word-sense induction method based on pre-trained masked language models (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged according to a corpus-derived sense inventory and where each sense is associated with indicative words. Evaluation on English Wikipedia that was sense-tagged using our method shows that both the induced senses, and the per-instance sense assignment, are of high quality even compared to WSD methods, such as Babelfy. Furthermore, by training a static word embeddings algorithm on the sense-tagged corpus, we obtain high-quality static senseful embeddings. These outperform existing senseful embeddings methods on the WiC dataset and on a new outlier detection dataset we developed. The data driven nature of the algorithm allows to induce corpora-specific senses, which may not appear in standard sense inventories, as we demonstrate using a case study on the scientific domain.