mlsanityData to 7 Oct 2026

English Word Sense Disambiguation in 2026: When the Labels Become the Bottleneck

arXiv 2609.17554 ↗First posted 19 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In English all-words word sense disambiguation (WSD), the labels, not the models, have become the bottleneck: frontier LLMs are accurate enough that the errors surviving in the gold standard decide benchmark rankings -- in the test sets we score on and, as we show causally, in the corpus we train on. We release lexEN, a WSD evaluation benchmark built as a conservative, human-adjudicated correction layer over Maru2022's ALLNEW benchmark (211 labels changed, 56 removed), and SenseBench, an auditable LLM WSD evaluation harness and living leaderboard (57 models, 192 runs). The task is inventory-constrained multiple choice (the model picks from the supplied WordNet senses), so the reported accuracies are a ceiling on what models achieve without that help. On lexEN-v1 the frontier LLMs converge near 95% (best, 95.6%), the top three families are statistically indistinguishable, and accuracy trades off against reasoning effort and cost across a ~2,500x price span. Relabeling SemCor with frontier models and retraining BEM, ESCHER, and ConSeC unchanged lifts them by several F1 points on test sets the relabeling never touched; we release the relabeled corpora and Glite LENS, a 298M bi-encoder trained on the repaired labels -- to our knowledge the strongest reported (83.6 Raganato ALL, 87.4 Maru ALLNEW) -- serving at ~0.13 per million items. On hard items, fine-grained WordNet senses are partly ill-posed even for experts (three-reviewer Fleiss kappa=0.537); coarsening raises annotator agreement and model accuracy together across four inventories, placing a top model inside the expert agreement band at coarse granularity (statistically equivalent under three of four) but significantly below it at fine. The binding constraint is now cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 71 references that are in the map. First posted on arXiv 19 Jul 2026.

Builds on

25 of 71 references.

References · 25

  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  2. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  4. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  5. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  6. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  7. 2020Are we done with ImageNet?cited by 329
  8. 2022Is GPT-3 a Good Data Annotator?cited by 208
  9. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  10. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  11. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
  12. 2020TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Taskcited by 96
13 more
  1. 2019GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledgecited by 78
  2. 2021Re-TACRED: Addressing Shortcomings of the TACRED Datasetcited by 73
  3. 2020Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoderscited by 61
  4. 2016Definition Modeling: Learning to define word embeddings in natural languagecited by 46
  5. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  6. 2019Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguationcited by 40
  7. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  8. 2025Exploring the Word Sense Disambiguation Capabilities of Large Language Modelscited by 10
  9. 2025Do Large Language Models Understand Word Senses?cited by 10
  10. 2025When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvementscited by 7
  11. 2024Coarse-Grained Sense Inventories Based on Semantic Matching between English Dictionariescited by 2
  12. 2025SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoccited by 2
  13. 2026Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026An Exploration-Analysis-Disambiguation Reasoning Framework for Word Sense Disambiguation with Low-Parameter LLMs
  2. 2026Ukrainian Visual Word Sense Disambiguation Benchmark
  3. 2023Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models
  4. 2024Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation
  5. 2023Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations
  6. 2026Bridging Lexical Ambiguity and Vision: A Mini Review on Visual Word Sense Disambiguation
  7. 2023Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
  8. 2023HKUST at SemEval-2023 Task 1: Visual Word Sense Disambiguation with Context Augmentation and Visual Assistance