mlsanityData to 14 Oct 2026

WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization

arXiv 2503.23779 ↗First posted 31 Mar 2025cs.CL cs.AIcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this study, we take a closer look at how Winograd schema challenges can be used to evaluate common sense reasoning in LLMs. Specifically, we evaluate generative models of different sizes on the popular WinoGrande benchmark. We release WinoWhat, a new corpus, in which each instance of the WinoGrande validation set is paraphrased. Additionally, we evaluate the performance on the challenge across five common sense knowledge categories, giving more fine-grained insights on what types of knowledge are more challenging for LLMs. Surprisingly, all models perform significantly worse on WinoWhat, implying that LLM reasoning capabilities are overestimated on WinoGrande. To verify whether this is an effect of benchmark memorization, we match benchmark instances to LLM trainingdata and create two test-suites. We observe that memorization has a minimal effect on model performance on WinoGrande.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 83 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

36 of 83 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction15 Oct
  2. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samples8 Nov
  3. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  10. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  11. 2018Gender Bias in Coreference Resolutioncited by 442
  12. 2018A Simple Method for Commonsense Reasoningcited by 223
21 more
  1. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  2. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  3. 2023What's In My Big Data?cited by 129
  4. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 123
  5. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  6. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 85
  7. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  8. 2021TIMEDIAL: Temporal Commonsense Reasoning in Dialogcited by 55
  9. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  10. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  11. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  12. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  13. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  14. 2022The Defeat of the Winograd Schema Challengecited by 25
  15. 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
  16. 2021Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understandingcited by 19
  17. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18
  18. 2020An Analysis of Dataset Overlap on Winograd-Style Taskscited by 16
  19. 2024WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Expertscited by 5
  20. 2024EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversariescited by 2
  21. 2021Semantic Categorization of Social Knowledge for Commonsense Question Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Findings of the Fourth Shared Task on Multilingual Coreference Resolution: Can LLMs Dethrone Traditional Approaches?cited by 6
  2. 2026Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  2. 2026MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
  3. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations
  4. 2026Benchmarking LLM Competence on Logical Inference over Probability Operators
  5. 2025Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation
  6. 2026Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
  7. 2025RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions
  8. 2024On Memorization of Large Language Models in Logical Reasoning