mlsanityData to 7 Oct 2026

EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries

arXiv 2402.13372 ↗First posted 20 Feb 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) excel at the Winograd Schema Challenge (WSC), a coreference resolution task testing common-sense reasoning through pronoun disambiguation, they struggle with instances that feature minor alterations or rewording. To address this, we introduce EvoGrad, an open-source platform that harnesses a human-in-the-loop approach to create a dynamic dataset tailored to such altered WSC instances. Leveraging ChatGPT's capabilities, we expand our task instances from 182 to 3,691, setting a new benchmark for diverse common-sense reasoning datasets. Additionally, we introduce the error depth metric, assessing model stability in dynamic tasks. Our results emphasize the challenge posed by EvoGrad: Even the best performing LLM, GPT-3.5, achieves an accuracy of 65.0% with an average error depth of 7.2, a stark contrast to human performance of 92. 8% accuracy without perturbation errors. This highlights ongoing model limitations and the value of dynamic datasets in uncovering them.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 44 of its 96 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

44 of 96 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
32 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  3. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  4. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  5. 2018Gender Bias in Coreference Resolutioncited by 438
  6. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  7. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  8. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  9. 2018Gender Bias in Neural Natural Language Processingcited by 210
  10. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  11. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  12. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  13. 2020Adversarial Filters of Dataset Biasescited by 159
  14. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  15. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  16. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  17. 2017A Continuously Growing Dataset of Sentential Paraphrasescited by 72
  18. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  19. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  20. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  21. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  22. 2022A Rationale-Centric Framework for Human-in-the-loop Machine Learningcited by 27
  23. 2016Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentationcited by 26
  24. 2021Substructure Substitution: Structured Data Augmentation for NLPcited by 25
  25. 2022The Defeat of the Winograd Schema Challengecited by 25
  26. 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
  27. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18
  28. 2020What's in a Name? Are BERT Named Entity Representations just as Good for any other Name?cited by 17
  29. 2020An Analysis of Dataset Overlap on Winograd-Style Taskscited by 16
  30. 2018Learning Visually-Grounded Semantics from Contrastive Adversarial Samplescited by 13
  31. 2021Studying Catastrophic Forgetting in Neural Ranking Modelscited by 9
  32. 2024WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Expertscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Language Model as Planner and Formalizer under Constraintscited by 3
  2. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2