# How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG

arXiv: https://arxiv.org/abs/1811.01778 · HTML: https://mlsanity.com/paper/1811.01778/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 18 of 18)

- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [A Systematic Investigation of Commonsense Knowledge in Large Language Models](https://mlsanity.com/paper/2111.00607/) (2021)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports](https://mlsanity.com/paper/2305.03598/) (2023)
- [Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?](https://mlsanity.com/paper/2404.03134/) (2024)
- [Are Prompt-based Models Clueless?](https://mlsanity.com/paper/2205.09295/) (2022)
- [Toward best research practices in AI Psychology](https://mlsanity.com/paper/2312.01276/) (2023)
- [Debiasing should be Good and Bad: Measuring the Consistency of Debiasing Techniques in Language Models](https://mlsanity.com/paper/2305.14307/) (2023)
- [Causal interventions expose implicit situation models for commonsense language understanding](https://mlsanity.com/paper/2306.03882/) (2023)
- [Who's Laughing Now? An Overview of Computational Humour Generation and Explanation](https://mlsanity.com/paper/2509.21175/) (2025)
- [Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models](https://mlsanity.com/paper/2109.05105/) (2021)
- [On Reality and the Limits of Language Data: Aligning LLMs with Human Norms](https://mlsanity.com/paper/2208.11981/) (2022)
- [EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries](https://mlsanity.com/paper/2402.13372/) (2024)
- [Ranking LLMs by compression](https://mlsanity.com/paper/2406.14171/) (2024)
- [WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization](https://mlsanity.com/paper/2503.23779/) (2025)
- [Attention-based Contrastive Learning for Winograd Schemas](https://mlsanity.com/paper/2109.05108/) (2021)
- [Problems and Countermeasures in Natural Language Processing Evaluation](https://mlsanity.com/paper/2104.09712/) (2021)
- [Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social Commonsense](https://mlsanity.com/paper/2105.05913/) (2021)

## Abstract (arXiv)

Recent studies have significantly improved the state-of-the-art on common-sense reasoning (CSR) benchmarks like the Winograd Schema Challenge (WSC) and SWAG. The question we ask in this paper is whether improved performance on these benchmarks represents genuine progress towards common-sense-enabled systems. We make case studies of both benchmarks and design protocols that clarify and qualify the results of previous work by analyzing threats to the validity of previous experimental designs. Our protocols account for several properties prevalent in common-sense benchmarks including size limitations, structural regularities, and variable instance difficulty.