# The Defeat of the Winograd Schema Challenge

arXiv: https://arxiv.org/abs/2201.02387 · HTML: https://mlsanity.com/paper/2201.02387/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 103 references that are in the map.
- First posted on arXiv 7 Jan 2022.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark](https://mlsanity.com/paper/2103.13009/) (2021)
- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://mlsanity.com/paper/1904.01172/) (2019)
- [RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2010.15925/) (2020)
- [Word meaning in minds and machines](https://mlsanity.com/paper/2008.01766/) (2020)
- [Generative Data Augmentation for Commonsense Reasoning](https://mlsanity.com/paper/2004.11546/) (2020)
- [WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge](https://mlsanity.com/paper/2005.05763/) (2020)
- [Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark](https://mlsanity.com/paper/1905.10425/) (2019)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [A Surprisingly Robust Trick for Winograd Schema Challenge](https://mlsanity.com/paper/1905.06290/) (2019)
- [Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models](https://mlsanity.com/paper/1908.06725/) (2019)
- [Gender Coreference and Bias Evaluation at WMT 2020](https://mlsanity.com/paper/2010.06018/) (2020)
- [Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup](https://mlsanity.com/paper/2112.06204/) (2021)
- [Attention Is (not) All You Need for Commonsense Reasoning](https://mlsanity.com/paper/1905.13497/) (2019)
- [WikiCREM: A Large Unsupervised Corpus for Coreference Resolution](https://mlsanity.com/paper/1908.08025/) (2019)
- [A Knowledge Hunting Framework for Common Sense Reasoning](https://mlsanity.com/paper/1810.01375/) (2018)
- [A Hybrid Neural Network Model for Commonsense Reasoning](https://mlsanity.com/paper/1907.11983/) (2019)
- [Precise Task Formalization Matters in Winograd Schema Evaluations](https://mlsanity.com/paper/2010.04043/) (2020)

## Built on by (most cited first, 25 of 25)

- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs](https://mlsanity.com/paper/2304.11164/) (2023)
- [KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices](https://mlsanity.com/paper/2601.21579/) (2026)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Causal interventions expose implicit situation models for commonsense language understanding](https://mlsanity.com/paper/2306.03882/) (2023)
- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [Transforming Agency. On the mode of existence of Large Language Models](https://mlsanity.com/paper/2407.10735/) (2024)
- [WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts](https://mlsanity.com/paper/2401.17703/) (2024)
- [Generalised Winograd Schema and its Contextuality](https://mlsanity.com/paper/2308.16498/) (2023)
- [Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge](https://mlsanity.com/paper/2405.16277/) (2024)
- [Sparse Relational Reasoning with Object-Centric Representations](https://mlsanity.com/paper/2207.07512/) (2022)
- [EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries](https://mlsanity.com/paper/2402.13372/) (2024)
- [WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization](https://mlsanity.com/paper/2503.23779/) (2025)
- [Developments in Sheaf-Theoretic Models of Natural Language Ambiguities](https://mlsanity.com/paper/2402.04505/) (2024)
- [Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution](https://mlsanity.com/paper/2410.09448/) (2024)
- [FLIP Reasoning Challenge](https://mlsanity.com/paper/2504.12256/) (2025)
- [Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study](https://mlsanity.com/paper/2506.13464/) (2025)
- [Pragmatic constraints and pronoun reference disambiguation: the possible and the impossible](https://mlsanity.com/paper/2204.01166/) (2022)
- [PronounFlow: A Hybrid Approach for Calibrating Pronouns in Sentences](https://mlsanity.com/paper/2308.15235/) (2023)
- [Reasoning about Ambiguous Definite Descriptions](https://mlsanity.com/paper/2310.14657/) (2023)
- [A Human-Machine Collaboration Framework for the Development of Schemas](https://mlsanity.com/paper/2402.07932/) (2024)
- [Natural, Artificial, and Human Intelligences](https://mlsanity.com/paper/2506.02183/) (2025)
- [A Community-driven vision for a new Knowledge Resource for AI](https://mlsanity.com/paper/2506.16596/) (2025)
- [Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation](https://mlsanity.com/paper/2511.17290/) (2025)
- [LOGOS-CA: A Cellular Automaton Using Natural Language as State and Rule](https://mlsanity.com/paper/2602.00036/) (2026)

## Abstract (arXiv)

The Winograd Schema Challenge - a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge - was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and discuss the lasting contributions of the flurry of research that has taken place on the WSC in the last decade. We discuss the significance of various datasets developed for WSC, and the research community's deeper understanding of the role of surrogate tasks in assessing the intelligence of an AI system.