mlsanityData to 7 Oct 2026

CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Models

arXiv 2605.12918 ↗First posted 13 May 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

To effectively interact with the real world, Large Language Models (LLMs) require entity-based commonsense reasoning, a challenging task that necessitates integrating factual knowledge about specific entities with commonsense inference. Existing datasets for evaluating LLM entity-based commonsense reasoning have largely focused on True/False or multiple-choice questions, leaving the explicit assessment of the model's ability in abductive reasoning about causes and effects and generating explanations largely unexamined. In this work, we introduce CommonWhy, a dataset of 15,000 why questions designed to evaluate entity-based commonsense reasoning about causal relationships in LLMs. CommonWhy also serves as a Knowledge Graph Question Answering (KGQA) benchmark, as all supporting knowledge required to answer its queries is available in the Wikidata knowledge graph. Unlike existing KGQA datasets, which primarily test fact retrieval, CommonWhy targets causal commonsense reasoning, establishing a new paradigm for KGQA evaluation. Experiments with state-of-the-art LLMs and LLM-based KGQA methods reveal their significant shortcomings, including frequent factual hallucinations and failures in causal reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 32 of its 50 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

32 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026A Balanced Neuro-Symbolic Approach for Commonsense Abductive Logic26 Jan
  2. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation3 Nov
  3. 2025CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge20 Apr
  4. 2024On Memorization of Large Language Models in Logical Reasoning30 Oct
  5. 2023CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks23 Oct
  6. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Language20 Dec
  7. 2022COPA-SSE: Semi-structured Explanations for Commonsense Reasoning18 Jan

Other references · 25

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2020Knowledge Graphscited by 619
  9. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  11. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  12. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
13 more
  1. 2020Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Basescited by 169
  2. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  3. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  4. 2023Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofittingcited by 70
  5. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  6. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  7. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21
  8. 2024Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answeringcited by 11
  9. 2024CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledgecited by 8
  10. 2020Personal Health Knowledge Graphs for Patientscited by 7
  11. 2017CommAI: Evaluating the first steps towards a useful general AIcited by 3
  12. 2023LogicRec: Recommendation with Users' Logical Requirementscited by 2
  13. 2023Bayesian Knowledge-driven Critiquing with Indirect Evidencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Causal Reasoning in Large Language Models: A Knowledge Graph Approach
  2. 2025Com2: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
  3. 2023CLadder: Assessing Causal Reasoning in Language Models
  4. 2024LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoning
  5. 2022CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm
  6. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
  7. 2024Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs
  8. 2025HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning