# Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders

arXiv: https://arxiv.org/abs/2005.02590 · HTML: https://mlsanity.com/paper/2005.02590/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2020.

## Built on by (most cited first, 30 of 61)

- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [A Survey on Contextualised Semantic Shift Detection](https://mlsanity.com/paper/2304.01666/) (2023)
- [Evaluating Distributional Distortion in Neural Language Modeling](https://mlsanity.com/paper/2203.12788/) (2022)
- [MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models](https://mlsanity.com/paper/2109.09237/) (2021)
- [ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD](https://mlsanity.com/paper/2205.09685/) (2022)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing](https://mlsanity.com/paper/2306.02052/) (2023)
- [FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary](https://mlsanity.com/paper/2102.07983/) (2021)
- [Construction of Knowledge Graphs: State and Challenges](https://mlsanity.com/paper/2302.11509/) (2023)
- [Efficient Zero-shot Event Extraction with Context-Definition Alignment](https://mlsanity.com/paper/2211.05156/) (2022)
- [SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks](https://mlsanity.com/paper/2310.19029/) (2023)
- [Do Large Language Models Understand Word Senses?](https://mlsanity.com/paper/2509.13905/) (2025)
- [What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories](https://mlsanity.com/paper/2302.03353/) (2023)
- [MadDog: A Web-based System for Acronym Identification and Disambiguation](https://mlsanity.com/paper/2101.09893/) (2021)
- [BERT-based Acronym Disambiguation with Multiple Training Strategies](https://mlsanity.com/paper/2103.00488/) (2021)
- [Richer Countries and Richer Representations](https://mlsanity.com/paper/2205.05093/) (2022)
- [Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles](https://mlsanity.com/paper/2406.06840/) (2024)
- [Incorporating Word Sense Disambiguation in Neural Language Models](https://mlsanity.com/paper/2106.07967/) (2021)
- [Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories](https://mlsanity.com/paper/2110.14091/) (2021)
- [Multilingual Word Sense Disambiguation with Unified Sense Representation](https://mlsanity.com/paper/2210.07447/) (2022)
- [Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information](https://mlsanity.com/paper/2305.01788/) (2023)
- [Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation](https://mlsanity.com/paper/2305.13119/) (2023)
- [Resolving Regular Polysemy in Named Entities](https://mlsanity.com/paper/2401.09758/) (2024)
- [Non-Parametric Few-Shot Learning for Word Sense Disambiguation](https://mlsanity.com/paper/2104.12677/) (2021)
- [Leveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation](https://mlsanity.com/paper/2107.00316/) (2021)
- [Patterns of Lexical Ambiguity in Contextualised Language Models](https://mlsanity.com/paper/2109.13032/) (2021)
- [A Survey on Semantic Processing Techniques](https://mlsanity.com/paper/2310.18345/) (2023)
- [Neural Sequence-to-Sequence Modeling with Attention by Leveraging Deep Learning Architectures for Enhanced Contextual Understanding in Abstractive Text Summarization](https://mlsanity.com/paper/2404.08685/) (2024)
- [Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense](https://mlsanity.com/paper/2410.21573/) (2024)
- [Word sense extension](https://mlsanity.com/paper/2306.05609/) (2023)

## Abstract (arXiv)

A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We propose a bi-encoder model that independently embeds (1) the target word with its surrounding context and (2) the dictionary definition, or gloss, of each sense. The encoders are jointly optimized in the same representation space, so that sense disambiguation can be performed by finding the nearest sense embedding for each target word embedding. Our system outperforms previous state-of-the-art models on English all-words WSD; these gains predominantly come from improved performance on rare senses, leading to a 31.1% error reduction on less frequent senses over prior work. This demonstrates that rare senses can be more effectively disambiguated by modeling their definitions.