# Towards better substitution-based word sense induction

arXiv: https://arxiv.org/abs/1905.12598 · HTML: https://mlsanity.com/paper/1905.12598/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 18 of 18)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [Characterizing English Variation across Social Media Communities with BERT](https://mlsanity.com/paper/2102.06820/) (2021)
- [Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution](https://mlsanity.com/paper/2206.11815/) (2022)
- [Large Scale Substitution-based Word Sense Induction](https://mlsanity.com/paper/2110.07681/) (2021)
- [RuDSI: graph-based word sense induction dataset for Russian](https://mlsanity.com/paper/2209.13750/) (2022)
- [PolyLM: Learning about Polysemy through Language Modeling](https://mlsanity.com/paper/2101.10448/) (2021)
- [Towards Automatic Construction of Filipino WordNet: Word Sense Induction and Synset Induction Using Sentence Embeddings](https://mlsanity.com/paper/2204.03251/) (2022)
- [Tracing Semantic Variation in Slang](https://mlsanity.com/paper/2210.08635/) (2022)
- [To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models](https://mlsanity.com/paper/2406.20054/) (2024)
- [Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection](https://mlsanity.com/paper/2106.07947/) (2021)
- [BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection](https://mlsanity.com/paper/2206.11865/) (2022)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [In the LLM era, Word Sense Induction remains unsolved](https://mlsanity.com/paper/2603.11686/) (2026)
- [Multilingual Substitution-based Word Sense Induction](https://mlsanity.com/paper/2405.11086/) (2024)
- [Text analysis and deep learning: A network approach](https://mlsanity.com/paper/2110.04151/) (2021)
- [Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization](https://mlsanity.com/paper/2210.05422/) (2022)
- [Deep-change at AXOLOTL-24: Orchestrating WSD and WSI Models for Semantic Change Modeling](https://mlsanity.com/paper/2408.05184/) (2024)

## Abstract (arXiv)

Word sense induction (WSI) is the task of unsupervised clustering of word usages within a sentence to distinguish senses. Recent work obtain strong results by clustering lexical substitutes derived from pre-trained RNN language models (ELMo). Adapting the method to BERT improves the scores even further. We extend the previous method to support a dynamic rather than a fixed number of clusters as supported by other prominent methods, and propose a method for interpreting the resulting clusters by associating them with their most informative substitutes. We then perform extensive error analysis revealing the remaining sources of errors in the WSI task. Our code is available at https://github.com/asafamr/bertwsi.