mlsanityData to 7 Oct 2026

The Defeat of the Winograd Schema Challenge

arXiv 2201.02387 ↗First posted 7 Jan 2022cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

The Winograd Schema Challenge - a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge - was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and discuss the lasting contributions of the flurry of research that has taken place on the WSC in the last decade. We discuss the significance of various datasets developed for WSC, and the research community's deeper understanding of the role of surrogate tasks in assessing the intelligence of an AI system.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 35 of its 103 references that are in the map. First posted on arXiv 7 Jan 2022.

Builds on

35 of 103 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  7. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  8. 2018Gender Bias in Coreference Resolutioncited by 438
  9. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  10. 2019Evaluating Gender Bias in Machine Translationcited by 247
  11. 2018A Simple Method for Commonsense Reasoningcited by 223
  12. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
23 more
  1. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  2. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  3. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 64
  4. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  5. 2020Word meaning in minds and machinescited by 51
  6. 2020Generative Data Augmentation for Commonsense Reasoningcited by 49
  7. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  8. 2019Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmarkcited by 32
  9. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  10. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  11. 2019Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Modelscited by 28
  12. 2020Gender Coreference and Bias Evaluation at WMT 2020cited by 21
  13. 2021Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setupcited by 16
  14. 2019Attention Is (not) All You Need for Commonsense Reasoningcited by 13
  15. 2019WikiCREM: A Large Unsupervised Corpus for Coreference Resolutioncited by 11
  16. 2018A Knowledge Hunting Framework for Common Sense Reasoningcited by 10
  17. 2019A Hybrid Neural Network Model for Commonsense Reasoningcited by 10
  18. 2020Precise Task Formalization Matters in Winograd Schema Evaluationscited by 7
  19. 2019Unsupervised Deep Structured Semantic Models for Commonsense Reasoningcited by 4
  20. 2019Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challengecited by 3
  21. 2019Using Answer Set Programming for Commonsense Reasoning in the Winograd Schema Challengecited by 2
  22. 2020Tackling Domain-Specific Winograd Schemas with Knowledge-Based Reasoning and Machine Learningcited by 2
  23. 2020Does the Objective Matter? Comparing Training Objectives for Pronoun Resolutioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 9; 2025: 6; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  2. 2023Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMscited by 20
  3. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  4. 2026KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matricescited by 10
  5. 2023Causal interventions expose implicit situation models for commonsense language understandingcited by 7
  6. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  7. 2024Transforming Agency. On the mode of existence of Large Language Modelscited by 7
  8. 2024WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Expertscited by 5
  9. 2023Generalised Winograd Schema and its Contextualitycited by 4
  10. 2024Picturing Ambiguity: A Visual Twist on the Winograd Schema Challengecited by 3
  11. 2022Sparse Relational Reasoning with Object-Centric Representationscited by 2
  12. 2024EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversariescited by 2
13 more
  1. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  2. 2024Developments in Sheaf-Theoretic Models of Natural Language Ambiguitiescited by 1
  3. 2024Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolutioncited by 1
  4. 2025FLIP Reasoning Challengecited by 1
  5. 2025Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Studycited by 1
  6. 2022Pragmatic constraints and pronoun reference disambiguation: the possible and the impossiblecited by 0
  7. 2023PronounFlow: A Hybrid Approach for Calibrating Pronouns in Sentencescited by 0
  8. 2023Reasoning about Ambiguous Definite Descriptionscited by 0
  9. 2024A Human-Machine Collaboration Framework for the Development of Schemascited by 0
  10. 2025Natural, Artificial, and Human Intelligencescited by 0
  11. 2025A Community-driven vision for a new Knowledge Resource for AIcited by 0
  12. 2025Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translationcited by 0
  13. 2026LOGOS-CA: A Cellular Automaton Using Natural Language as State and Rulecited by 0