# What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories

arXiv: https://arxiv.org/abs/2302.03353 · HTML: https://mlsanity.com/paper/2302.03353/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 76 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.03659/) (2021)
- [Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders](https://mlsanity.com/paper/2005.02590/) (2020)
- [Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start](https://mlsanity.com/paper/2010.02584/) (2020)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System](https://mlsanity.com/paper/2104.11882/) (2021)
- [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning](https://mlsanity.com/paper/2205.01376/) (2022)
- [Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations](https://mlsanity.com/paper/1910.00194/) (2019)
- [Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference](https://mlsanity.com/paper/2202.06167/) (2022)
- [Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences](https://mlsanity.com/paper/2009.11795/) (2020)
- [Ask2Transformers: Zero-Shot Domain labelling with Pre-trained Language Models](https://mlsanity.com/paper/2101.02661/) (2021)

## Built on by (most cited first, 9 of 9)

- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset](https://mlsanity.com/paper/2306.11167/) (2023)
- [Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation](https://mlsanity.com/paper/2411.18337/) (2024)
- [In the LLM era, Word Sense Induction remains unsolved](https://mlsanity.com/paper/2603.11686/) (2026)
- [LLMRS: Unlocking Potentials of LLM-Based Recommender Systems for Software Purchase](https://mlsanity.com/paper/2401.06676/) (2024)
- [Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition](https://mlsanity.com/paper/2406.14894/) (2024)
- [Prompt Balance Matters: Understanding How Imbalanced Few-Shot Learning Affects Multilingual Sense Disambiguation in LLMs](https://mlsanity.com/paper/2510.03762/) (2025)
- [DACE For Railway Acronym Disambiguation](https://mlsanity.com/paper/2512.18357/) (2025)
- [SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation](https://mlsanity.com/paper/2604.16262/) (2026)

## Abstract (arXiv)

Language Models are the core for almost any Natural Language Processing system nowadays. One of their particularities is their contextualized representations, a game changer feature when a disambiguation between word senses is necessary. In this paper we aim to explore to what extent language models are capable of discerning among senses at inference time. We performed this analysis by prompting commonly used Languages Models such as BERT or RoBERTa to perform the task of Word Sense Disambiguation (WSD). We leverage the relation between word senses and domains, and cast WSD as a textual entailment problem, where the different hypothesis refer to the domains of the word senses. Our results show that this approach is indeed effective, close to supervised systems.