# ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2502.15487 · HTML: https://mlsanity.com/paper/2502.15487/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [Joint Reasoning for Temporal and Causal Relations](https://mlsanity.com/paper/1906.04941/) (2019)
- [e-CARE: a New Dataset for Exploring Explainable Causal Reasoning](https://mlsanity.com/paper/2205.05849/) (2022)
- [CRAB: Assessing the Strength of Causal Relationships Between Real-world Events](https://mlsanity.com/paper/2311.04284/) (2023)
- [Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models](https://mlsanity.com/paper/2403.14859/) (2024)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuning](https://mlsanity.com/paper/2401.00139/) (2023)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment](https://mlsanity.com/paper/2402.13956/) (2024)
- [Predicting Directionality in Causal Relations in Text](https://mlsanity.com/paper/2103.13606/) (2021)

## Built on by (most cited first, 3 of 3)

- [Do Large Language Models Show Biases in Causal Learning? Insights from Contingency Judgment](https://mlsanity.com/paper/2510.13985/) (2025)
- [InferenceEvolve: Towards Automated Causal Effect Estimators through Self-Evolving AI](https://mlsanity.com/paper/2604.04274/) (2026)
- [DeepImagine: Clinical Trial Outcome Prediction via Stepwise Local Counterfactual Imaginations](https://mlsanity.com/paper/2604.23054/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in tasks requiring interpretive and inferential accuracy. In this paper, we introduce ExpliCa, a new dataset for evaluating LLMs in explicit causal reasoning. ExpliCa uniquely integrates both causal and temporal relations presented in different linguistic orders and explicitly expressed by linguistic connectives. The dataset is enriched with crowdsourced human acceptability ratings. We tested LLMs on ExpliCa through prompting and perplexity-based metrics. We assessed seven commercial and open-source LLMs, revealing that even top models struggle to reach 0.80 accuracy. Interestingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events. Finally, perplexity-based scores and prompting performance are differently affected by model size.