mlsanityData to 14 Oct 2026

WinoDict: Probing language models for in-context word acquisition

arXiv 2209.12153 ↗First posted 25 Sep 2022cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

We introduce a new in-context learning paradigm to measure Large Language Models' (LLMs) ability to learn novel words during inference. In particular, we rewrite Winograd-style co-reference resolution problems by replacing the key concept word with a synthetic but plausible word that the model must understand to complete the task. Solving this task requires the model to make use of the dictionary definition of the new word given in the prompt. This benchmark addresses word acquisition, one important aspect of the diachronic degradation known to afflict LLMs. As LLMs are frozen in time at the moment they are trained, they are normally unable to reflect the way language changes over time. We show that the accuracy of LLMs compared to the original Winograd tasks decreases radically in our benchmark, thus identifying a limitation of current models and providing a benchmark to measure future improvements in LLMs ability to do in-context learning.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 25 Sep 2022.

Builds on

18 of 30 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  10. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  11. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  12. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
6 more
  1. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  2. 2018A Simple Method for Commonsense Reasoningcited by 223
  3. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  4. 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 41
  5. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  6. 2015Learning to Understand Phrases by Embedding the Dictionarycited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 10; 2026: 2.

15 papers.

Sort:

By area:

  1. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 18
  2. 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Taskscited by 8
  3. 2025Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemiacited by 8
  4. 2023MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretationscited by 7
  5. 2025Rapid Word Learning Through Meta In-Context Learningcited by 4
  6. 2026Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidencecited by 3
  7. 2023MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisitioncited by 2
  8. 2025SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoccited by 2
  9. 2025WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectivescited by 2
  10. 2025Plant in Cupboard, Orange on Rably, Inat Aphone. Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environmentcited by 1
  11. 2025LLMs Learn Constructions That Humans Do Not Knowcited by 1
  12. 2025Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contextscited by 0
3 more
  1. 2025Towards Contamination Resistant Benchmarkscited by 0
  2. 2025Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priorscited by 0
  3. 2026KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?cited by 0