mlsanityData to 7 Oct 2026

WinoGrande: An Adversarial Winograd Schema Challenge at Scale

arXiv 1907.10641 ↗First posted 24 Jul 2019cs.CLcited by 194

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4-79.1%, which are 15-35% below human performance of 94.0%, depending on the amount of the training data allowed. Furthermore, we establish new state-of-the-art results on five related benchmarks - WSC (90.1%), DPR (93.1%), COPA (90.6%), KnowRef (85.6%), and Winogender (97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 24; 2023: 28; 2024: 43; 2025: 38; 2026: 35.

194 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023BloombergGPT: A Large Language Model for Financecited by 923
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2023The Falcon Series of Open Language Modelscited by 526
  12. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
38 more
  1. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  2. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  3. 2022UL2: Unifying Language Learning Paradigmscited by 292
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  6. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  7. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  8. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  9. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  10. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  11. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  12. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  13. 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
  14. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  15. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  16. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  17. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  18. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  19. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  20. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  21. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  22. 2021DialFact: A Benchmark for Fact-Checking in Dialoguecited by 62
  23. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  24. 2023NormBank: A Knowledge Bank of Situational Social Normscited by 60
  25. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 58
  26. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  27. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  28. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  29. 2021Prompting Contrastive Explanations for Commonsense Reasoning Taskscited by 45
  30. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  31. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  32. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  33. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  34. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  35. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  36. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  37. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  38. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33

50 most cited of 194 All 194 (JSON)