# Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contexts

arXiv: https://arxiv.org/abs/2601.15479 · HTML: https://mlsanity.com/paper/2601.15479/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://mlsanity.com/paper/2211.07342/) (2022)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?](https://mlsanity.com/paper/2506.21215/) (2025)
- [CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models](https://mlsanity.com/paper/2112.11941/) (2021)
- [Causal Inference with Large Language Model: A Survey](https://mlsanity.com/paper/2409.09822/) (2024)
- [Financial Document Causality Detection Shared Task (FinCausal 2020)](https://mlsanity.com/paper/2012.02505/) (2020)
- [Causality for Large Language Models](https://mlsanity.com/paper/2410.15319/) (2024)
- [Causality Extraction based on Self-Attentive BiLSTM-CRF with Transferred Embeddings](https://mlsanity.com/paper/1904.07629/) (2019)
- [Large Language Models for Causal Discovery: Current Landscape and Future Directions](https://mlsanity.com/paper/2402.11068/) (2024)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [Document-level Causal Relation Extraction with Knowledge-guided Binary Question Answering](https://mlsanity.com/paper/2410.04752/) (2024)
- [Event Causality Is Key to Computational Story Understanding](https://mlsanity.com/paper/2311.09648/) (2023)
- [Domain Adaptative Causality Encoder](https://mlsanity.com/paper/2011.13549/) (2020)
- [Causality extraction from medical text using Large Language Models (LLMs)](https://mlsanity.com/paper/2407.10020/) (2024)
- [BoschAI @ Causal News Corpus 2023: Robust Cause-Effect Span Extraction using Multi-Layer Sequence Tagging and Data Augmentation](https://mlsanity.com/paper/2312.06338/) (2023)
- [Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis](https://mlsanity.com/paper/2402.14484/) (2024)

## Built on by (most cited first, 1 of 1)

- [PubMedCausal: A Span-Level Annotated Corpus for Causal Relation Extraction in Biomedical Text](https://mlsanity.com/paper/2605.28363/) (2026)

## Abstract (arXiv)

The safe deployment of large language models (LLMs) in high-stakes fields like biomedicine, requires them to be able to reason about cause and effect. We investigate this ability by testing 13 open-source LLMs on a fundamental task: pairwise causal discovery (PCD) from text. Our benchmark, using 12 diverse datasets, evaluates two core skills: 1) Causal Detection (identifying if a text contains a causal link) and 2) Causal Extraction (pulling out the exact cause and effect phrases). We tested various prompting methods, from simple instructions (zero-shot) to more complex strategies like Chain-of-Thought (CoT) and Few-shot In-Context Learning (FICL). The results show major deficiencies in current models. The best model for detection, DeepSeek-R1-Distill-Llama-70B, only achieved a mean score of 49.57% (Cdetect), while the best for extraction, Qwen2.5-Coder-32B-Instruct, reached just 47.12% (Cextract). Models performed best on simple, explicit, single-sentence relations. However, performance plummeted for more difficult (and realistic) cases, such as implicit relationships, links spanning multiple sentences, and texts containing multiple causal pairs. We provide a unified evaluation framework, built on a dataset validated with high inter-annotator agreement (kappa ≥ 0.758), and make all our data, code, and prompts publicly available to spur further research. hrefhttps://github.com/sydneyanuyah/CausalDiscoveryCode available here: https://github.com/sydneyanuyah/CausalDiscovery