mlsanityData to 7 Oct 2026

CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

arXiv 2112.11941 ↗First posted 22 Dec 2021cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 35 of its 72 references that are in the map. First posted on arXiv 22 Dec 2021.

Builds on

35 of 72 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
23 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  8. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  9. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  10. 2021Towards Causal Representation Learningcited by 207
  11. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  12. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  13. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  14. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  15. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  16. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
  17. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  18. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  19. 2017A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turkcited by 38
  20. 2020Critical Thinking for Language Modelscited by 31
  21. 2019CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoningcited by 12
  22. 2020An Analysis of the Adaptation Speed of Causal Modelscited by 9
  23. 2019A Structured Distributional Model of Sentence Meaning and Processingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 13; 2025: 7; 2026: 2.

33 papers.

Sort:

By area:

  1. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  3. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  4. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  5. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  6. 2023Integrating Large Language Model for Improved Causal Discoverycited by 54
  7. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 46
  8. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  9. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarioscited by 27
  10. 2023CRAB: Assessing the Strength of Causal Relationships Between Real-world Eventscited by 27
  11. 2024Causality for Large Language Modelscited by 24
  12. 2022CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generationcited by 23
21 more
  1. 2024ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understandingcited by 23
  2. 2025On Robustness and Reliability of Benchmark-Based Evaluation of LLMscited by 23
  3. 2023Causal Structure Learning Supervised by Large Language Modelcited by 21
  4. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalitiescited by 18
  5. 2024Causal Evaluation of Language Modelscited by 18
  6. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  7. 2024Reasoning Elicitation in Language Models via Counterfactual Feedbackcited by 14
  8. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Modelscited by 13
  9. 2024Failure Modes of LLMs for Causal Reasoning on Narrativescited by 12
  10. 2023Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challengescited by 9
  11. 2023Learning to Plan with Natural Languagecited by 6
  12. 2024RNR: Teaching Large Language Models to Follow Roles and Rulescited by 6
  13. 2025CYCLE-INSTRUCT: Fully Seed-Free Instruction Tuning via Dual Self-Training and Cycle Consistencycited by 6
  14. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Studycited by 5
  15. 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
  16. 2025Exploring the Hidden Reasoning Process of Large Language Models by Misleading Themcited by 3
  17. 2024ACCORD: Closing the Commonsense Measurability Gapcited by 1
  18. 2025A Study of Commonsense Reasoning over Visual Object Propertiescited by 1
  19. 2025How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Datasetcited by 1
  20. 2026Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contextscited by 1
  21. 2026CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungscited by 1