# To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models

arXiv: https://arxiv.org/abs/2406.20054 · HTML: https://mlsanity.com/paper/2406.20054/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Survey of Computational Approaches to Lexical Semantic Change](https://mlsanity.com/paper/1811.06278/) (2018)
- [UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection](https://mlsanity.com/paper/2005.00050/) (2020)
- [Capturing Evolution in Word Usage: Just Add More Clusters?](https://mlsanity.com/paper/2001.06629/) (2020)
- [Towards better substitution-based word sense induction](https://mlsanity.com/paper/1905.12598/) (2019)
- [Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge](https://mlsanity.com/paper/2010.13057/) (2020)
- [A Benchmark and Scoring Algorithm for Enriching Arabic Synonyms](https://mlsanity.com/paper/2302.02232/) (2023)
- [Large Scale Substitution-based Word Sense Induction](https://mlsanity.com/paper/2110.07681/) (2021)
- [Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case](https://mlsanity.com/paper/2109.01935/) (2021)
- [Towards Automatic Construction of Filipino WordNet: Word Sense Induction and Synset Induction Using Sentence Embeddings](https://mlsanity.com/paper/2204.03251/) (2022)

## Built on by (most cited first, 3 of 3)

- [In the LLM era, Word Sense Induction remains unsolved](https://mlsanity.com/paper/2603.11686/) (2026)
- [CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation](https://mlsanity.com/paper/2508.04494/) (2025)
- [Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models](https://mlsanity.com/paper/2609.05333/) (2026)

## Abstract (arXiv)

Polysemy and synonymy are two crucial interrelated facets of lexical ambiguity. While both phenomena are widely documented in lexical resources and have been studied extensively in NLP, leading to dedicated systems, they are often being considered independently in practical problems. While many tasks dealing with polysemy (e.g. Word Sense Disambiguation or Induction) highlight the role of word's senses, the study of synonymy is rooted in the study of concepts, i.e. meanings shared across the lexicon. In this paper, we introduce Concept Induction, the unsupervised task of learning a soft clustering among words that defines a set of concepts directly from data. This task generalizes Word Sense Induction. We propose a bi-level approach to Concept Induction that leverages both a local lemma-centric view and a global cross-lexicon view to induce concepts. We evaluate the obtained clustering on SemCor's annotated data and obtain good performance (BCubed F1 above 0.60). We find that the local and the global levels are mutually beneficial to induce concepts and also senses in our setting. Finally, we create static embeddings representing our induced concepts and use them on the Word-in-Context task, obtaining competitive performance with the State-of-the-Art.