# WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge

arXiv: https://arxiv.org/abs/2005.05763 · HTML: https://mlsanity.com/paper/2005.05763/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 30 of 43)

- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning](https://mlsanity.com/paper/2104.07644/) (2021)
- [Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?](https://mlsanity.com/paper/2104.10809/) (2021)
- [Prompting Contrastive Explanations for Commonsense Reasoning Tasks](https://mlsanity.com/paper/2106.06823/) (2021)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models](https://mlsanity.com/paper/2112.11941/) (2021)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [On the Diversity and Limits of Human Explanations](https://mlsanity.com/paper/2106.11988/) (2021)
- [Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?](https://mlsanity.com/paper/2402.11597/) (2024)
- [Dimensions of Commonsense Knowledge](https://mlsanity.com/paper/2101.04640/) (2021)
- [The Defeat of the Winograd Schema Challenge](https://mlsanity.com/paper/2201.02387/) (2022)
- [METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation](https://mlsanity.com/paper/2205.02593/) (2022)
- [ALERT: Adapting Language Models to Reasoning Tasks](https://mlsanity.com/paper/2212.08286/) (2022)
- [Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup](https://mlsanity.com/paper/2112.06204/) (2021)
- [multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning](https://mlsanity.com/paper/2106.01354/) (2021)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [A Critical Review of Causal Reasoning Benchmarks for Large Language Models](https://mlsanity.com/paper/2407.08029/) (2024)
- [More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives](https://mlsanity.com/paper/2501.04070/) (2025)
- [Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations](https://mlsanity.com/paper/2305.14618/) (2023)
- [Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks](https://mlsanity.com/paper/2104.10193/) (2021)
- [On Evaluating Explanation Utility for Human-AI Decision Making in NLP](https://mlsanity.com/paper/2407.03545/) (2024)
- [Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations](https://mlsanity.com/paper/2211.07517/) (2022)
- [Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation](https://mlsanity.com/paper/2304.12443/) (2023)
- [HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervision](https://mlsanity.com/paper/2305.14237/) (2023)
- [WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts](https://mlsanity.com/paper/2401.17703/) (2024)
- [Explanation Regeneration via Information Bottleneck](https://mlsanity.com/paper/2212.09603/) (2022)
- [Generalised Winograd Schema and its Contextuality](https://mlsanity.com/paper/2308.16498/) (2023)
- [Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction](https://mlsanity.com/paper/2410.12040/) (2024)

## Abstract (arXiv)

In this paper, we present the first comprehensive categorization of essential commonsense knowledge for answering the Winograd Schema Challenge (WSC). For each of the questions, we invite annotators to first provide reasons for making correct decisions and then categorize them into six major knowledge categories. By doing so, we better understand the limitation of existing methods (i.e., what kind of knowledge cannot be effectively represented or inferred with existing methods) and shed some light on the commonsense knowledge that we need to acquire in the future for better commonsense reasoning. Moreover, to investigate whether current WSC models can understand the commonsense or they simply solve the WSC questions based on the statistical bias of the dataset, we leverage the collected reasons to develop a new task called WinoWhy, which requires models to distinguish plausible reasons from very similar but wrong reasons for all WSC questions. Experimental results prove that even though pre-trained language representation models have achieved promising progress on the original WSC dataset, they are still struggling at WinoWhy. Further experiments show that even though supervised models can achieve better performance, the performance of these models can be sensitive to the dataset distribution. WinoWhy and all codes are available at: https://github.com/HKUST-KnowComp/WinoWhy.