mlsanityData to 13 Oct 2026

Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction

arXiv 2410.12040 ↗First posted 15 Oct 2024cs.CL cs.AIcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

While Large Language Models (LLMs) have showcased remarkable proficiency in reasoning, there is still a concern about hallucinations and unreliable reasoning issues due to semantic associations and superficial logical chains. To evaluate the extent to which LLMs perform robust reasoning instead of relying on superficial logical chains, we propose a new evaluation dataset, the Concept-Reversed Winograd Schema Challenge (CR-WSC), based on the famous Winograd Schema Challenge (WSC) dataset. By simply reversing the concepts to those that are more associated with the wrong answer, we find that the performance of LLMs drops significantly despite the rationale of reasoning remaining the same. Furthermore, we propose Abstraction-of-Thought (AoT), a novel prompt method for recovering adversarial cases to normal cases using conceptual abstraction to improve LLMs' robustness and consistency in reasoning, as demonstrated by experiments on CR-WSC.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 54 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

31 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Conceptual and Unbiased Reasoning in Language Models30 Mar
  2. 2023CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering24 May
  3. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May

Other references · 28

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,117
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,954
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,594
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,404
  12. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
16 more
  1. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  2. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 377
  3. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 168
  4. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 158
  5. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  6. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  7. 2021Entailment as Few-Shot Learnercited by 122
  8. 2023GraphLLM: Boosting Graph Reasoning Ability of Large Language Modelcited by 106
  9. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  10. 2024Can GNN be Good Adapter for LLMs?cited by 71
  11. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  12. 2021Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resourcescited by 30
  13. 2024CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoningcited by 28
  14. 2023Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?cited by 26
  15. 2023AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graphcited by 17
  16. 2024AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimationcited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertaintycited by 10
  2. 2025Policy Frameworks for Transparent Chain-of-Thought Reasoning in Large Language Modelscited by 4
  3. 2025WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorizationcited by 2
  4. 2026HarnessBridge: Learnable Bidirectional Controller for LLM Agent Harnesscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models
  2. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  3. 2025Towards Reasoning Ability of Small Language Models
  4. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2026Large Language Model Reasoning Failures
  7. 2026Robust Reasoning Benchmark
  8. 2025CER: Confidence Enhanced Reasoning in LLMs