mlsanityData to 14 Oct 2026

WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts

arXiv 2401.17703 ↗First posted 31 Jan 2024cs.CL cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

The Winograd Schema Challenge (WSC) serves as a prominent benchmark for evaluating machine understanding. While Large Language Models (LLMs) excel at answering WSC questions, their ability to generate such questions remains less explored. In this work, we propose Tree-of-Experts (ToE), a novel prompting method which enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods). Using this approach, we introduce WSC+, a novel dataset comprising 3,026 LLM-generated sentences. Notably, we extend the WSC framework by incorporating new 'ambiguous' and 'offensive' categories, providing a deeper insight into model overconfidence and bias. Our analysis reveals nuances in generation-evaluation consistency, suggesting that LLMs may not always outperform in evaluating their own generated questions when compared to those crafted by other models. On WSC+, GPT-4, the top-performing LLM, achieves an accuracy of 68.7%, significantly below the human benchmark of 95.1%.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

27 of 39 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  8. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  9. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  10. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  11. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  12. 2023Language Models can Solve Computer Taskscited by 372
15 more
  1. 2018A Simple Method for Commonsense Reasoningcited by 223
  2. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  3. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  4. 2021Generating Datasets with Pretrained Language Modelscited by 193
  5. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  6. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  7. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  8. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  9. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  10. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  11. 2021Substructure Substitution: Structured Data Augmentation for NLPcited by 25
  12. 2022The Defeat of the Winograd Schema Challengecited by 25
  13. 2018A Knowledge Hunting Framework for Common Sense Reasoningcited by 10
  14. 2022Making Pretrained Language Models Good Long-tailed Learnerscited by 8
  15. 2021Alleviating the Knowledge-Language Inconsistency: A Study for Deep Commonsense Knowledgecited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Picturing Ambiguity: A Visual Twist on the Winograd Schema Challengecited by 3
  2. 2024EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversariescited by 2
  3. 2025The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theoriescited by 2
  4. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  5. 2024Detecting Bias and Enhancing Diagnostic Accuracy in Large Language Models for Healthcarecited by 1