# CRAB: Assessing the Strength of Causal Relationships Between Real-world Events

arXiv: https://arxiv.org/abs/2311.04284 · HTML: https://mlsanity.com/paper/2311.04284/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Multi-Sentence Argument Linking](https://mlsanity.com/paper/1911.03766/) (2019)
- [Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs](https://mlsanity.com/paper/1905.07374/) (2019)
- [Understanding Causality with Large Language Models: Feasibility and Opportunities](https://mlsanity.com/paper/2304.05524/) (2023)
- [Causal Discovery Toolbox: Uncover causal relationships in Python](https://mlsanity.com/paper/1903.02278/) (2019)
- [CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models](https://mlsanity.com/paper/2112.11941/) (2021)
- [Distinguishing cause from effect using observational data: methods and benchmarks](https://mlsanity.com/paper/1412.3773/) (2014)
- [SciLens: Evaluating the Quality of Scientific News Articles Using Social Media and Scientific Literature Indicators](https://mlsanity.com/paper/1903.05538/) (2019)

## Built on by (most cited first, 28 of 28)

- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?](https://mlsanity.com/paper/2506.21215/) (2025)
- [Large Language Models and Causal Inference in Collaboration: A Survey](https://mlsanity.com/paper/2403.09606/) (2024)
- [Causality for Large Language Models](https://mlsanity.com/paper/2410.15319/) (2024)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [Agentic AI Needs a Systems Theory](https://mlsanity.com/paper/2503.00237/) (2025)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [On the Reliability of Large Language Models for Causal Discovery](https://mlsanity.com/paper/2407.19638/) (2024)
- [CHECKWHY: Causal Fact Verification via Argument Structure](https://mlsanity.com/paper/2408.10918/) (2024)
- [C2P: Featuring Large Language Models with Causal Reasoning](https://mlsanity.com/paper/2407.18069/) (2024)
- [WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts](https://mlsanity.com/paper/2506.15594/) (2025)
- [Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities](https://mlsanity.com/paper/2408.08105/) (2024)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists](https://mlsanity.com/paper/2605.26029/) (2026)
- [ACCESS : A Benchmark for Abstract Causal Event Discovery and Reasoning](https://mlsanity.com/paper/2502.08148/) (2025)
- [ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models](https://mlsanity.com/paper/2502.15487/) (2025)
- [Large Language Models for Zero-shot Inference of Causal Structures in Biology](https://mlsanity.com/paper/2503.04347/) (2025)
- [Causal Graph based Event Reasoning using Semantic Relation Experts](https://mlsanity.com/paper/2506.06910/) (2025)
- [Investigating Counterclaims in Causality Extraction from Text](https://mlsanity.com/paper/2510.08224/) (2025)
- [WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning](https://mlsanity.com/paper/2606.11816/) (2026)
- [From Adoption to Adaption: Tracing the Diffusion of New Emojis on Twitter](https://mlsanity.com/paper/2402.14187/) (2024)
- [ClimateCause: Complex and Implicit Causal Structures in Climate Reports](https://mlsanity.com/paper/2604.14856/) (2026)
- [Causal Discovery Inspired Unsupervised Domain Adaptation for Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2406.15490/) (2024)
- [SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models](https://mlsanity.com/paper/2603.21720/) (2026)
- [Abstract Event Causal Rules: Induction and Application](https://mlsanity.com/paper/2608.05205/) (2026)
- [Contextual Causality with Large Language Models: A Survey](https://mlsanity.com/paper/2609.22409/) (2026)
- [Evaluating Whether LLMs Can Reliably Connect the DOTs?](https://mlsanity.com/paper/2609.38406/) (2026)
- [CoBe: A Benchmark For Conversational Counterfactual Text Editing](https://mlsanity.com/paper/2610.13057/) (2026)

## Abstract (arXiv)

Understanding narratives requires reasoning about the cause-and-effect relationships between events mentioned in the text. While existing foundation models yield impressive results in many NLP tasks requiring reasoning, it is unclear whether they understand the complexity of the underlying network of causal relationships of events in narratives. In this work, we present CRAB, a new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives. CRAB contains fine-grained, contextual causality annotations for ~2.7K pairs of real-world events that describe various newsworthy event timelines (e.g., the acquisition of Twitter by Elon Musk). Using CRAB, we measure the performance of several large language models, demonstrating that most systems achieve poor performance on the task. Motivated by classical causal principles, we also analyze the causal structures of groups of events in CRAB, and find that models perform worse on causal reasoning when events are derived from complex causal structures compared to simple linear causal chains. We make our dataset and code available to the research community.