# English Word Sense Disambiguation in 2026: When the Labels Become the Bottleneck

arXiv: https://arxiv.org/abs/2609.17554 · HTML: https://mlsanity.com/paper/2609.17554/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 71 references that are in the map.
- First posted on arXiv 19 Jul 2026.

## Builds on (most cited first, 25 of 25)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [Re-TACRED: Addressing Shortcomings of the TACRED Dataset](https://mlsanity.com/paper/2104.08398/) (2021)
- [Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders](https://mlsanity.com/paper/2005.02590/) (2020)
- [Definition Modeling: Learning to define word embeddings in natural language](https://mlsanity.com/paper/1612.00394/) (2016)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://mlsanity.com/paper/1906.10007/) (2019)
- [Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance](https://mlsanity.com/paper/2410.18889/) (2024)
- [Exploring the Word Sense Disambiguation Capabilities of Large Language Models](https://mlsanity.com/paper/2503.08662/) (2025)
- [Do Large Language Models Understand Word Senses?](https://mlsanity.com/paper/2509.13905/) (2025)
- [When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvements](https://mlsanity.com/paper/2511.19794/) (2025)
- [Coarse-Grained Sense Inventories Based on Semantic Matching between English Dictionaries](https://mlsanity.com/paper/2409.06386/) (2024)
- [SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoc](https://mlsanity.com/paper/2503.05958/) (2025)
- [Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents](https://mlsanity.com/paper/2606.27416/) (2026)

## Abstract (arXiv)

In English all-words word sense disambiguation (WSD), the labels, not the models, have become the bottleneck: frontier LLMs are accurate enough that the errors surviving in the gold standard decide benchmark rankings -- in the test sets we score on and, as we show causally, in the corpus we train on. We release lexEN, a WSD evaluation benchmark built as a conservative, human-adjudicated correction layer over Maru2022's ALLNEW benchmark (211 labels changed, 56 removed), and SenseBench, an auditable LLM WSD evaluation harness and living leaderboard (57 models, 192 runs). The task is inventory-constrained multiple choice (the model picks from the supplied WordNet senses), so the reported accuracies are a ceiling on what models achieve without that help. On lexEN-v1 the frontier LLMs converge near 95% (best, 95.6%), the top three families are statistically indistinguishable, and accuracy trades off against reasoning effort and cost across a ~2,500x price span. Relabeling SemCor with frontier models and retraining BEM, ESCHER, and ConSeC unchanged lifts them by several F1 points on test sets the relabeling never touched; we release the relabeled corpora and Glite LENS, a 298M bi-encoder trained on the repaired labels -- to our knowledge the strongest reported (83.6 Raganato ALL, 87.4 Maru ALLNEW) -- serving at ~0.13 per million items. On hard items, fine-grained WordNet senses are partly ill-posed even for experts (three-reviewer Fleiss kappa=0.537); coarsening raises annotator agreement and model accuracy together across four inventories, placing a top model inside the expert agreement band at coarse granularity (statistically equivalent under three of four) but significantly below it at fine. The binding constraint is now cost.