mlsanityData to 7 Oct 2026

Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema

arXiv 2104.08161 ↗First posted 16 Apr 2021cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

The Winograd Schema (WS) has been proposed as a test for measuring commonsense capabilities of models. Recently, pre-trained language model-based approaches have boosted performance on some WS benchmarks but the source of improvement is still not clear. This paper suggests that the apparent progress on WS may not necessarily reflect progress in commonsense reasoning. To support this claim, we first show that the current evaluation method of WS is sub-optimal and propose a modification that uses twin sentences for evaluation. We also propose two new baselines that indicate the existence of artifacts in WS benchmarks. We then develop a method for evaluating WS-like sentences in a zero-shot setting to account for the commonsense reasoning abilities acquired during the pretraining and observe that popular language models perform randomly in this setting when using our more strict evaluation. We conclude that the observed progress is mostly due to the use of supervision in training WS models, which is not likely to successfully support all the required commonsense reasoning skills and knowledge.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 46 of its 72 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

46 of 72 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  11. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  12. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
34 more
  1. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  3. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  4. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  5. 2018A Simple Method for Commonsense Reasoningcited by 223
  6. 2019Assessing BERT's Syntactic Abilitiescited by 191
  7. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  8. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  9. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  10. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  11. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  12. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  13. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  14. 2019ASER: A Large-scale Eventuality Knowledge Graphcited by 89
  15. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 86
  16. 2018Event2Mind: Commonsense Inference on Events, Intents, and Reactionscited by 82
  17. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  18. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  19. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  20. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  21. 2020A Review of Winograd Schema Challenge Datasets and Approachescited by 27
  22. 2017Verb Physics: Relative Physical Knowledge of Actions and Objectscited by 26
  23. 2019How Large Are Lions? Inducing Distributions over Quantitative Attributescited by 26
  24. 2020Contrastive Self-Supervised Learning for Commonsense Reasoningcited by 25
  25. 2016Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objectscited by 22
  26. 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
  27. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18
  28. 2020An Analysis of Dataset Overlap on Winograd-Style Taskscited by 16
  29. 2019Attention Is (not) All You Need for Commonsense Reasoningcited by 13
  30. 2018A Knowledge Hunting Framework for Common Sense Reasoningcited by 10
  31. 2019Solving Hard Coreference Problemscited by 9
  32. 2019SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisitioncited by 8
  33. 2020Precise Task Formalization Matters in Winograd Schema Evaluationscited by 7
  34. 2019Using Answer Set Programming for Commonsense Reasoning in the Winograd Schema Challengecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 8; 2024: 6; 2025: 3; 2026: 1.

32 papers.

Sort:

By area:

  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  4. 2022On the Paradox of Learning to Reason from Datacited by 93
  5. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  6. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  7. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  8. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  9. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  10. 2024Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Provingcited by 35
  11. 2022The Defeat of the Winograd Schema Challengecited by 25
  12. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
20 more
  1. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Modelscited by 18
  2. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  3. 2023Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanationscited by 11
  4. 2024WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Casecited by 9
  5. 2022Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learningcited by 8
  6. 2022longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attackscited by 8
  7. 2023Toward best research practices in AI Psychologycited by 8
  8. 2023Causal interventions expose implicit situation models for commonsense language understandingcited by 7
  9. 2023CLOMO: Counterfactual Logical Modification with Large Language Modelscited by 7
  10. 2024Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistentlycited by 3
  11. 2024Picturing Ambiguity: A Visual Twist on the Winograd Schema Challengecited by 3
  12. 2022CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigmcited by 2
  13. 2024EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversariescited by 2
  14. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  15. 2026Caliper: Probing Lexical Anchors versus Causal Structure in LLMscited by 2
  16. 2022A Theoretically Grounded Benchmark for Evaluating Machine Commonsensecited by 1
  17. 2022Validity Assessment of Legal Will Statements as Natural Language Inferencecited by 1
  18. 2025MASS: Overcoming Language Bias in Image-Text Matchingcited by 1
  19. 2023PronounFlow: A Hybrid Approach for Calibrating Pronouns in Sentencescited by 0
  20. 2025Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translationcited by 0